add testing and plan milestones

2026-01-16 19:50:59 +01:00
parent 25b0603913
commit 2e097e4e54
8 changed files with 2540 additions and 28 deletions
--- a/MILESTONE-2.md
+++ b/MILESTONE-2.md
@@ -0,0 +1,186 @@
+# Milestone 2: Testing Framework
+
+**Status**: 95% Complete
+**Goal**: Automated UI testing for rapid iteration and AI agent verification.
+
+---
+
+## Overview
+
+The testing framework enables automated validation of UI behavior through:
+- UI hierarchy inspection (JSON dump)
+- Input simulation and recording
+- Log verification for navigation events
+- Visual regression testing via screenshot diff
+- JSON test results compatible with CI/CD
+
+---
+
+## Current State
+
+### Completed Components
+
+| Component | File | Status |
+|-----------|------|--------|
+| Action Types | `designer/src/testing/action_types.h` | Complete |
+| Action Recorder | `designer/src/testing/action_recorder.h/.cpp` | Complete |
+| Action Player | `designer/src/testing/action_player.h/.cpp` | Complete |
+| UI Inspector | `designer/src/testing/ui_inspector.cpp` | Complete |
+| Visual Capture | `designer/src/testing/visual_capture.cpp` | Complete |
+| Test Runner | `designer-test/src/test_runner.cpp` | Complete |
+| CLI Integration | `designer/src/main.cpp` | Complete |
+
+### Already Working
+
+1. **Action Types** (`action_types.h`)
+   - TapAction, SwipeAction, LongPressAction
+   - ButtonAction, WaitAction, KeyAction
+   - ActionSequence container with metadata
+
+2. **Action Recorder** (`action_recorder.h/.cpp`)
+   - Mouse down/up/move tracking
+   - Automatic gesture classification (tap vs swipe vs long press)
+   - JSON serialization/deserialization
+   - Configurable thresholds
+
+3. **Action Player** (`action_player.h/.cpp`)
+   - Timestamp-based playback
+   - RmlUi event injection
+   - Pause/resume/stop controls
+   - Progress tracking
+
+4. **UI Inspector** (`ui_inspector.cpp`)
+   - Full element tree traversal
+   - JSON export with bounds, IDs, classes
+   - Continuous hierarchy dumping mode
+
+5. **Test Runner** (`designer-test/`)
+   - WindowController (Windows SendInput)
+   - HierarchyReader (element lookup)
+   - LogParser (navigation verification)
+   - All 5 navigation tests passing
+
+---
+
+## Recently Completed
+
+### Screenshot Diff Implementation (Complete)
+
+**File**: `designer/src/testing/visual_capture.cpp`
+
+Implemented real PNG pixel-by-pixel comparison:
+- `LoadPNG()` helper function loads PNG files using libpng
+- `PixelsMatch()` compares pixels with configurable tolerance (default: 2)
+- `CompareImages()` returns difference ratio (0.0 = identical, 1.0 = completely different)
+
+### Recording/Playback CLI (Complete)
+
+**File**: `designer/src/main.cpp`
+
+Added CLI options and keyboard controls:
+- `--record <file>` - Enable recording mode
+- `--playback <file>` - Play back recorded actions
+- F5 - Start/stop recording (saves to specified file)
+- F6 - Pause/resume playback
+
+**Usage**:
+```bash
+# Record interactions
+mosis-designer.exe home.rml --record my-test.json
+# Press F5 to start recording, interact, press F5 again to save
+
+# Play back
+mosis-designer.exe home.rml --playback my-test.json
+```
+
+---
+
+## Remaining Task
+
+### Task 2.3: GLFW Input Hooks for Recording
+
+**Status**: Partially Complete
+**Effort**: Medium
+**Limitation**: Requires RmlUi Backend modification
+
+**Current State**:
+- Recording infrastructure is complete (ActionRecorder)
+- Playback works fully (ActionPlayer calls RmlUi context directly)
+- CLI and keyboard controls are wired up
+- **Missing**: Direct GLFW callback access for mouse recording
+
+**The Problem**:
+The RmlUi Backend abstraction handles all GLFW callbacks internally and doesn't expose them for interception. To record actual mouse events, we would need to either:
+
+1. **Modify RmlUi Backend** (third-party code)
+   - Add callback hooks to `RmlUi_Backend_GLFW_GL3.cpp`
+   - Expose GLFW window handle for custom callbacks
+
+2. **Fork RmlUi Backends** (more maintainable)
+   - Copy Backend files into designer project
+   - Add recording hooks
+
+3. **Alternative: Element-Based Recording**
+   - Listen to RmlUi events after processing
+   - Record element clicks by ID rather than coordinates
+   - Less precise but avoids backend modification
+
+**Workaround for Now**:
+Tests can be created manually by:
+1. Using the UI hierarchy to find element coordinates
+2. Writing JSON test files directly
+3. Using the external designer-test framework (Windows SendInput)
+
+**Future Work**:
+Consider option 2 (fork backends) when recording becomes a priority.
+
+---
+
+## File Changes Summary
+
+| File | Changes | Status |
+|------|---------|--------|
+| `designer/src/testing/visual_capture.cpp` | Real PNG pixel comparison | Done |
+| `designer/src/main.cpp` | --record/--playback CLI, F5/F6 keys | Done |
+| `designer/src/RmlUi_Backend.h` | Expose GLFW window for recording | Future |
+
+---
+
+## Test Recording Format
+
+```json
+{
+  "name": "Navigate to contacts and back",
+  "description": "Test navigation flow",
+  "screen_width": 540,
+  "screen_height": 960,
+  "initial_screen": "apps/home/home.rml",
+  "actions": [
+    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
+    {"type": "wait", "duration": 1000, "timestamp": 100},
+    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
+    {"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
+  ]
+}
+```
+
+---
+
+## Acceptance Criteria
+
+- [x] `CompareImages()` returns accurate pixel difference ratio
+- [ ] `--record <file>` captures all mouse/key events to JSON (needs GLFW hooks)
+- [x] `--playback <file>` replays recorded actions with correct timing
+- [x] Recording stops gracefully on F5 or window close
+- [x] Playback shows progress in console
+- [x] Screenshot diff with 2-pixel tolerance per channel
+
+---
+
+## Future Enhancements (Not This Milestone)
+
+- Visual diff output (highlight changed pixels)
+- Parallel test execution
+- Android action recording/playback
+- Cross-platform test runner
+- Coverage reporting