MosisService/docs/MILESTONE-2.md

Milestone 2: Testing Framework

Status: 100% Complete Goal: Automated UI testing for rapid iteration and AI agent verification.

---

Overview

The testing framework enables automated validation of UI behavior through:

• UI hierarchy inspection (JSON dump)
• Input simulation and recording
• Log verification for navigation events
• Visual regression testing via screenshot diff
• JSON test results compatible with CI/CD

---

Current State

Completed Components

Component	File	Status
Action Types	designer/src/testing/action_types.h	Complete
Action Recorder	designer/src/testing/action_recorder.h/.cpp	Complete
Action Player	designer/src/testing/action_player.h/.cpp	Complete
UI Inspector	designer/src/testing/ui_inspector.cpp	Complete
Visual Capture	designer/src/testing/visual_capture.cpp	Complete
Test Runner	designer-test/src/test_runner.cpp	Complete
CLI Integration	designer/src/main.cpp	Complete

Already Working

1. Action Types (action_types.h)

   • TapAction, SwipeAction, LongPressAction
   • ButtonAction, WaitAction, KeyAction
   • ActionSequence container with metadata

2. Action Recorder (action_recorder.h/.cpp)

   • Mouse down/up/move tracking
   • Automatic gesture classification (tap vs swipe vs long press)
   • JSON serialization/deserialization
   • Configurable thresholds

3. Action Player (action_player.h/.cpp)

   • Timestamp-based playback
   • RmlUi event injection
   • Pause/resume/stop controls
   • Progress tracking

4. UI Inspector (ui_inspector.cpp)

   • Full element tree traversal
   • JSON export with bounds, IDs, classes
   • Continuous hierarchy dumping mode

5. Test Runner (designer-test/)

   • WindowController (Windows SendInput)
   • HierarchyReader (element lookup)
   • LogParser (navigation verification)
   • All 5 navigation tests passing

---

Recently Completed

Screenshot Diff Implementation (Complete)

File: designer/src/testing/visual_capture.cpp

Implemented real PNG pixel-by-pixel comparison:

• LoadPNG() helper function loads PNG files using libpng
• PixelsMatch() compares pixels with configurable tolerance (default: 2)
• CompareImages() returns difference ratio (0.0 = identical, 1.0 = completely different)

Recording/Playback CLI (Complete)

File: designer/src/main.cpp

Added CLI options and keyboard controls:

• --record <file> - Enable recording mode
• --playback <file> - Play back recorded actions
• F5 - Start/stop recording (saves to specified file)
• F6 - Pause/resume playback

Usage:

# Record interactions
mosis-designer.exe home.rml --record my-test.json
# Press F5 to start recording, interact, press F5 again to save

# Play back
mosis-designer.exe home.rml --playback my-test.json

---

Completed: GLFW Input Hooks for Recording

Task 2.3: GLFW Input Hooks

Status: Complete

Solution Implemented: Forked the RmlUi backend files into designer/src/backend/ with input recording hooks:

1. RmlUi_Backend.h - Added callback type definitions:

   • MouseButtonCallback - Called on mouse button press/release
   • MouseMoveCallback - Called on mouse movement
   • KeyCallback - Called on key press/release

2. RmlUi_Backend_GLFW_GL3.cpp - Modified GLFW callbacks to:

   • Track mouse position in framebuffer coordinates
   • Call recording callbacks before forwarding to RmlUi
   • Support all three callback types

3. main.cpp - Connected callbacks to ActionRecorder:

   • Mouse button events trigger RecordMouseDown/RecordMouseUp
   • Mouse move events trigger RecordMouseMove
   • Key events trigger RecordKey

Files Added:

• designer/src/backend/RmlUi_Backend.h
• designer/src/backend/RmlUi_Backend_GLFW_GL3.cpp
• designer/src/backend/RmlUi_Platform_GLFW.h
• designer/src/backend/RmlUi_Platform_GLFW.cpp

---

File Changes Summary

File	Changes	Status
designer/src/testing/visual_capture.cpp	Real PNG pixel comparison	Done
designer/src/main.cpp	--record/--playback CLI, F5/F6 keys	Done
designer/src/RmlUi_Backend.h	Expose GLFW window for recording	Future

---

Test Recording Format

{
  "name": "Navigate to contacts and back",
  "description": "Test navigation flow",
  "screen_width": 540,
  "screen_height": 960,
  "initial_screen": "apps/home/home.rml",
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
  ]
}

---

Acceptance Criteria

• CompareImages() returns accurate pixel difference ratio
• --record <file> captures all mouse/key events to JSON (needs GLFW hooks)
• --playback <file> replays recorded actions with correct timing
• Recording stops gracefully on F5 or window close
• Playback shows progress in console
• Screenshot diff with 2-pixel tolerance per channel

---

Future Enhancements (Not This Milestone)

• Visual diff output (highlight changed pixels)
• Parallel test execution
• Android action recording/playback
• Cross-platform test runner
• Coverage reporting