MosisService/MILESTONE-2.md

Milestone 2: Testing Framework

Status: 95% Complete Goal: Automated UI testing for rapid iteration and AI agent verification.

---

Overview

The testing framework enables automated validation of UI behavior through:

• UI hierarchy inspection (JSON dump)
• Input simulation and recording
• Log verification for navigation events
• Visual regression testing via screenshot diff
• JSON test results compatible with CI/CD

---

Current State

Completed Components

Component	File	Status
Action Types	designer/src/testing/action_types.h	Complete
Action Recorder	designer/src/testing/action_recorder.h/.cpp	Complete
Action Player	designer/src/testing/action_player.h/.cpp	Complete
UI Inspector	designer/src/testing/ui_inspector.cpp	Complete
Visual Capture	designer/src/testing/visual_capture.cpp	Complete
Test Runner	designer-test/src/test_runner.cpp	Complete
CLI Integration	designer/src/main.cpp	Complete

Already Working

1. Action Types (action_types.h)

   • TapAction, SwipeAction, LongPressAction
   • ButtonAction, WaitAction, KeyAction
   • ActionSequence container with metadata

2. Action Recorder (action_recorder.h/.cpp)

   • Mouse down/up/move tracking
   • Automatic gesture classification (tap vs swipe vs long press)
   • JSON serialization/deserialization
   • Configurable thresholds

3. Action Player (action_player.h/.cpp)

   • Timestamp-based playback
   • RmlUi event injection
   • Pause/resume/stop controls
   • Progress tracking

4. UI Inspector (ui_inspector.cpp)

   • Full element tree traversal
   • JSON export with bounds, IDs, classes
   • Continuous hierarchy dumping mode

5. Test Runner (designer-test/)

   • WindowController (Windows SendInput)
   • HierarchyReader (element lookup)
   • LogParser (navigation verification)
   • All 5 navigation tests passing

---

Recently Completed

Screenshot Diff Implementation (Complete)

File: designer/src/testing/visual_capture.cpp

Implemented real PNG pixel-by-pixel comparison:

• LoadPNG() helper function loads PNG files using libpng
• PixelsMatch() compares pixels with configurable tolerance (default: 2)
• CompareImages() returns difference ratio (0.0 = identical, 1.0 = completely different)

Recording/Playback CLI (Complete)

File: designer/src/main.cpp

Added CLI options and keyboard controls:

• --record <file> - Enable recording mode
• --playback <file> - Play back recorded actions
• F5 - Start/stop recording (saves to specified file)
• F6 - Pause/resume playback

Usage:

# Record interactions
mosis-designer.exe home.rml --record my-test.json
# Press F5 to start recording, interact, press F5 again to save

# Play back
mosis-designer.exe home.rml --playback my-test.json

---

Remaining Task

Task 2.3: GLFW Input Hooks for Recording

Status: Partially Complete Effort: Medium Limitation: Requires RmlUi Backend modification

Current State:

• Recording infrastructure is complete (ActionRecorder)
• Playback works fully (ActionPlayer calls RmlUi context directly)
• CLI and keyboard controls are wired up
• Missing: Direct GLFW callback access for mouse recording

The Problem: The RmlUi Backend abstraction handles all GLFW callbacks internally and doesn't expose them for interception. To record actual mouse events, we would need to either:

1. Modify RmlUi Backend (third-party code)

   • Add callback hooks to RmlUi_Backend_GLFW_GL3.cpp
   • Expose GLFW window handle for custom callbacks

2. Fork RmlUi Backends (more maintainable)

   • Copy Backend files into designer project
   • Add recording hooks

3. Alternative: Element-Based Recording

   • Listen to RmlUi events after processing
   • Record element clicks by ID rather than coordinates
   • Less precise but avoids backend modification

Workaround for Now: Tests can be created manually by:

1. Using the UI hierarchy to find element coordinates
2. Writing JSON test files directly
3. Using the external designer-test framework (Windows SendInput)

Future Work: Consider option 2 (fork backends) when recording becomes a priority.

---

File Changes Summary

File	Changes	Status
designer/src/testing/visual_capture.cpp	Real PNG pixel comparison	Done
designer/src/main.cpp	--record/--playback CLI, F5/F6 keys	Done
designer/src/RmlUi_Backend.h	Expose GLFW window for recording	Future

---

Test Recording Format

{
  "name": "Navigate to contacts and back",
  "description": "Test navigation flow",
  "screen_width": 540,
  "screen_height": 960,
  "initial_screen": "apps/home/home.rml",
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
  ]
}

---

Acceptance Criteria

• CompareImages() returns accurate pixel difference ratio
• --record <file> captures all mouse/key events to JSON (needs GLFW hooks)
• --playback <file> replays recorded actions with correct timing
• Recording stops gracefully on F5 or window close
• Playback shows progress in console
• Screenshot diff with 2-pixel tolerance per channel

---

Future Enhancements (Not This Milestone)

• Visual diff output (highlight changed pixels)
• Parallel test execution
• Android action recording/playback
• Cross-platform test runner
• Coverage reporting