187 lines
5.6 KiB
Markdown
187 lines
5.6 KiB
Markdown
# Milestone 2: Testing Framework
|
|
|
|
**Status**: 95% Complete
|
|
**Goal**: Automated UI testing for rapid iteration and AI agent verification.
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
The testing framework enables automated validation of UI behavior through:
|
|
- UI hierarchy inspection (JSON dump)
|
|
- Input simulation and recording
|
|
- Log verification for navigation events
|
|
- Visual regression testing via screenshot diff
|
|
- JSON test results compatible with CI/CD
|
|
|
|
---
|
|
|
|
## Current State
|
|
|
|
### Completed Components
|
|
|
|
| Component | File | Status |
|
|
|-----------|------|--------|
|
|
| Action Types | `designer/src/testing/action_types.h` | Complete |
|
|
| Action Recorder | `designer/src/testing/action_recorder.h/.cpp` | Complete |
|
|
| Action Player | `designer/src/testing/action_player.h/.cpp` | Complete |
|
|
| UI Inspector | `designer/src/testing/ui_inspector.cpp` | Complete |
|
|
| Visual Capture | `designer/src/testing/visual_capture.cpp` | Complete |
|
|
| Test Runner | `designer-test/src/test_runner.cpp` | Complete |
|
|
| CLI Integration | `designer/src/main.cpp` | Complete |
|
|
|
|
### Already Working
|
|
|
|
1. **Action Types** (`action_types.h`)
|
|
- TapAction, SwipeAction, LongPressAction
|
|
- ButtonAction, WaitAction, KeyAction
|
|
- ActionSequence container with metadata
|
|
|
|
2. **Action Recorder** (`action_recorder.h/.cpp`)
|
|
- Mouse down/up/move tracking
|
|
- Automatic gesture classification (tap vs swipe vs long press)
|
|
- JSON serialization/deserialization
|
|
- Configurable thresholds
|
|
|
|
3. **Action Player** (`action_player.h/.cpp`)
|
|
- Timestamp-based playback
|
|
- RmlUi event injection
|
|
- Pause/resume/stop controls
|
|
- Progress tracking
|
|
|
|
4. **UI Inspector** (`ui_inspector.cpp`)
|
|
- Full element tree traversal
|
|
- JSON export with bounds, IDs, classes
|
|
- Continuous hierarchy dumping mode
|
|
|
|
5. **Test Runner** (`designer-test/`)
|
|
- WindowController (Windows SendInput)
|
|
- HierarchyReader (element lookup)
|
|
- LogParser (navigation verification)
|
|
- All 5 navigation tests passing
|
|
|
|
---
|
|
|
|
## Recently Completed
|
|
|
|
### Screenshot Diff Implementation (Complete)
|
|
|
|
**File**: `designer/src/testing/visual_capture.cpp`
|
|
|
|
Implemented real PNG pixel-by-pixel comparison:
|
|
- `LoadPNG()` helper function loads PNG files using libpng
|
|
- `PixelsMatch()` compares pixels with configurable tolerance (default: 2)
|
|
- `CompareImages()` returns difference ratio (0.0 = identical, 1.0 = completely different)
|
|
|
|
### Recording/Playback CLI (Complete)
|
|
|
|
**File**: `designer/src/main.cpp`
|
|
|
|
Added CLI options and keyboard controls:
|
|
- `--record <file>` - Enable recording mode
|
|
- `--playback <file>` - Play back recorded actions
|
|
- F5 - Start/stop recording (saves to specified file)
|
|
- F6 - Pause/resume playback
|
|
|
|
**Usage**:
|
|
```bash
|
|
# Record interactions
|
|
mosis-designer.exe home.rml --record my-test.json
|
|
# Press F5 to start recording, interact, press F5 again to save
|
|
|
|
# Play back
|
|
mosis-designer.exe home.rml --playback my-test.json
|
|
```
|
|
|
|
---
|
|
|
|
## Remaining Task
|
|
|
|
### Task 2.3: GLFW Input Hooks for Recording
|
|
|
|
**Status**: Partially Complete
|
|
**Effort**: Medium
|
|
**Limitation**: Requires RmlUi Backend modification
|
|
|
|
**Current State**:
|
|
- Recording infrastructure is complete (ActionRecorder)
|
|
- Playback works fully (ActionPlayer calls RmlUi context directly)
|
|
- CLI and keyboard controls are wired up
|
|
- **Missing**: Direct GLFW callback access for mouse recording
|
|
|
|
**The Problem**:
|
|
The RmlUi Backend abstraction handles all GLFW callbacks internally and doesn't expose them for interception. To record actual mouse events, we would need to either:
|
|
|
|
1. **Modify RmlUi Backend** (third-party code)
|
|
- Add callback hooks to `RmlUi_Backend_GLFW_GL3.cpp`
|
|
- Expose GLFW window handle for custom callbacks
|
|
|
|
2. **Fork RmlUi Backends** (more maintainable)
|
|
- Copy Backend files into designer project
|
|
- Add recording hooks
|
|
|
|
3. **Alternative: Element-Based Recording**
|
|
- Listen to RmlUi events after processing
|
|
- Record element clicks by ID rather than coordinates
|
|
- Less precise but avoids backend modification
|
|
|
|
**Workaround for Now**:
|
|
Tests can be created manually by:
|
|
1. Using the UI hierarchy to find element coordinates
|
|
2. Writing JSON test files directly
|
|
3. Using the external designer-test framework (Windows SendInput)
|
|
|
|
**Future Work**:
|
|
Consider option 2 (fork backends) when recording becomes a priority.
|
|
|
|
---
|
|
|
|
## File Changes Summary
|
|
|
|
| File | Changes | Status |
|
|
|------|---------|--------|
|
|
| `designer/src/testing/visual_capture.cpp` | Real PNG pixel comparison | Done |
|
|
| `designer/src/main.cpp` | --record/--playback CLI, F5/F6 keys | Done |
|
|
| `designer/src/RmlUi_Backend.h` | Expose GLFW window for recording | Future |
|
|
|
|
---
|
|
|
|
## Test Recording Format
|
|
|
|
```json
|
|
{
|
|
"name": "Navigate to contacts and back",
|
|
"description": "Test navigation flow",
|
|
"screen_width": 540,
|
|
"screen_height": 960,
|
|
"initial_screen": "apps/home/home.rml",
|
|
"actions": [
|
|
{"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
|
|
{"type": "wait", "duration": 1000, "timestamp": 100},
|
|
{"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
|
|
{"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] `CompareImages()` returns accurate pixel difference ratio
|
|
- [ ] `--record <file>` captures all mouse/key events to JSON (needs GLFW hooks)
|
|
- [x] `--playback <file>` replays recorded actions with correct timing
|
|
- [x] Recording stops gracefully on F5 or window close
|
|
- [x] Playback shows progress in console
|
|
- [x] Screenshot diff with 2-pixel tolerance per channel
|
|
|
|
---
|
|
|
|
## Future Enhancements (Not This Milestone)
|
|
|
|
- Visual diff output (highlight changed pixels)
|
|
- Parallel test execution
|
|
- Android action recording/playback
|
|
- Cross-platform test runner
|
|
- Coverage reporting
|