finish the testing framework
This commit is contained in:
145
TESTING.md
145
TESTING.md
@@ -10,6 +10,8 @@ The testing framework enables automated validation of UI behavior through:
|
||||
2. **Input Simulation**: Mouse clicks via Windows SendInput API
|
||||
3. **Log Verification**: Check navigation events via log file parsing
|
||||
4. **Test Results**: JSON output compatible with CI/CD pipelines
|
||||
5. **Action Recording/Playback**: Record and replay user interactions
|
||||
6. **Visual Regression**: Screenshot comparison with pixel-level diff
|
||||
|
||||
## Components
|
||||
|
||||
@@ -21,8 +23,22 @@ The desktop designer serves as the test target. When launched with testing optio
|
||||
mosis-designer.exe home.rml --log test.log --hierarchy hierarchy.json
|
||||
```
|
||||
|
||||
- `--log`: Writes all RmlUi INFO logs to file (navigation events, errors)
|
||||
- `--hierarchy`: Dumps UI element tree to JSON each frame
|
||||
**Testing Options**:
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--log FILE` | Write all RmlUi INFO logs to file (navigation events, errors) |
|
||||
| `--hierarchy FILE` | Dump UI element tree to JSON each frame |
|
||||
| `--record FILE` | Enable action recording mode (F5 to start/stop) |
|
||||
| `--playback FILE` | Play back recorded actions from JSON file |
|
||||
|
||||
**Keyboard Controls**:
|
||||
|
||||
| Key | Function |
|
||||
|-----|----------|
|
||||
| F5 | Start/stop recording (when `--record` is enabled) |
|
||||
| F6 | Pause/resume playback (when `--playback` is enabled) |
|
||||
| F12 | Take screenshot (saves to current directory) |
|
||||
|
||||
### Test Runner (designer-test.exe)
|
||||
|
||||
@@ -231,6 +247,124 @@ results.SaveToFile(resultsPath);
|
||||
}
|
||||
```
|
||||
|
||||
## Action Recording and Playback
|
||||
|
||||
The designer supports recording user interactions and playing them back for automated testing.
|
||||
|
||||
### Recording Actions
|
||||
|
||||
```bash
|
||||
# Start designer with recording enabled
|
||||
mosis-designer.exe home.rml --record my-test.json
|
||||
|
||||
# Press F5 to start recording
|
||||
# Interact with the UI (clicks, swipes, etc.)
|
||||
# Press F5 again to stop and save
|
||||
```
|
||||
|
||||
Recording is automatically saved when you close the window.
|
||||
|
||||
### Playing Back Actions
|
||||
|
||||
```bash
|
||||
# Play back a recorded test
|
||||
mosis-designer.exe home.rml --playback my-test.json
|
||||
```
|
||||
|
||||
Use F6 to pause/resume playback.
|
||||
|
||||
### Action Recording Format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Navigate to contacts",
|
||||
"description": "Test navigation flow",
|
||||
"screen_width": 540,
|
||||
"screen_height": 960,
|
||||
"initial_screen": "apps/home/home.rml",
|
||||
"actions": [
|
||||
{"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
|
||||
{"type": "wait", "duration": 1000, "timestamp": 100},
|
||||
{"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
|
||||
{"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Supported Action Types
|
||||
|
||||
| Type | Fields | Description |
|
||||
|------|--------|-------------|
|
||||
| `tap` | x, y, timestamp | Single tap at coordinates |
|
||||
| `swipe` | x1, y1, x2, y2, duration, timestamp | Swipe gesture |
|
||||
| `long_press` | x, y, duration, timestamp | Long press gesture |
|
||||
| `button` | button, timestamp | Hardware button ("back", "home") |
|
||||
| `wait` | duration, timestamp | Pause between actions |
|
||||
| `key` | key_code, pressed, timestamp | Keyboard input |
|
||||
|
||||
### Creating Test Files Manually
|
||||
|
||||
You can also create test files manually using the UI hierarchy to find element coordinates:
|
||||
|
||||
```bash
|
||||
# Get element coordinates from hierarchy
|
||||
mosis-designer.exe home.rml --hierarchy hierarchy.json
|
||||
# Read hierarchy.json to find element bounds
|
||||
# Write action JSON with those coordinates
|
||||
```
|
||||
|
||||
## Screenshot Comparison
|
||||
|
||||
The testing framework includes pixel-level screenshot comparison for visual regression testing.
|
||||
|
||||
### Using Screenshot Comparison
|
||||
|
||||
```cpp
|
||||
#include "testing/visual_capture.h"
|
||||
|
||||
// Capture a screenshot
|
||||
mosis::testing::VisualCapture capture(540, 960);
|
||||
capture.CaptureScreenshot("current.png");
|
||||
|
||||
// Compare two screenshots
|
||||
float diff = mosis::testing::VisualCapture::CompareImages("baseline.png", "current.png");
|
||||
|
||||
// diff = 0.0 means identical
|
||||
// diff = 1.0 means completely different
|
||||
// Typical threshold: diff < 0.01 (less than 1% different)
|
||||
```
|
||||
|
||||
### Comparison Details
|
||||
|
||||
- Compares RGBA pixels with a tolerance of 2 per channel
|
||||
- Returns ratio of differing pixels (0.0 to 1.0)
|
||||
- Different dimensions = 1.0 (completely different)
|
||||
- Missing files = 1.0 (comparison failed)
|
||||
|
||||
### Visual Regression Test Example
|
||||
|
||||
```cpp
|
||||
bool TestVisualRegression(TestContext& ctx) {
|
||||
// Navigate to screen
|
||||
GoHome(ctx);
|
||||
ClickById(ctx, "dock-phone");
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
|
||||
|
||||
// Capture screenshot
|
||||
mosis::testing::VisualCapture capture(ctx.width, ctx.height);
|
||||
capture.CaptureScreenshot("dialer-current.png");
|
||||
|
||||
// Compare with baseline
|
||||
float diff = mosis::testing::VisualCapture::CompareImages(
|
||||
"baselines/dialer-expected.png",
|
||||
"dialer-current.png"
|
||||
);
|
||||
|
||||
// Allow up to 1% difference
|
||||
return diff < 0.01f;
|
||||
}
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Command Line
|
||||
@@ -389,9 +523,12 @@ class NavigationTest {
|
||||
|
||||
## Future Improvements
|
||||
|
||||
- [ ] Action recording (capture user interactions)
|
||||
- [ ] Screenshot comparison (visual regression testing)
|
||||
- [x] Action recording (capture user interactions) - *CLI and infrastructure complete*
|
||||
- [x] Screenshot comparison (visual regression testing) - *Pixel-level diff implemented*
|
||||
- [x] Action playback with timing - *Fully functional*
|
||||
- [x] GLFW input hooks for automatic mouse recording - *Complete via forked backend*
|
||||
- [ ] Android instrumentation test suite
|
||||
- [ ] Parallel test execution
|
||||
- [ ] Test coverage reporting
|
||||
- [ ] Cross-platform test runner (desktop + Android)
|
||||
- [ ] Visual diff output (highlight changed pixels)
|
||||
|
||||
Reference in New Issue
Block a user