omar/MosisService

Fork 0

Files

omigamedev 010e11cf6b move docs to docs/ folder, merge architecture files, update references

2026-01-19 09:02:11 +01:00

14 KiB

Raw Blame History

Mosis UI Testing Framework

This document describes the automated UI testing framework for Mosis virtual smartphone.

Overview

The testing framework enables automated validation of UI behavior through:

UI Hierarchy Inspection: JSON dump of all UI elements with bounds
Input Simulation: Mouse clicks via Windows SendInput API
Log Verification: Check navigation events via log file parsing
Test Results: JSON output compatible with CI/CD pipelines
Action Recording/Playback: Record and replay user interactions
Visual Regression: Screenshot comparison with pixel-level diff

Components

Designer (mosis-designer.exe)

The desktop designer serves as the test target. When launched with testing options:

mosis-designer.exe home.rml --log test.log --hierarchy hierarchy.json

Testing Options:

Option	Description
`--log FILE`	Write all RmlUi INFO logs to file (navigation events, errors)
`--hierarchy FILE`	Dump UI element tree to JSON each frame
`--record FILE`	Enable action recording mode (F5 to start/stop)
`--playback FILE`	Play back recorded actions from JSON file

Keyboard Controls:

Key	Function
F5	Start/stop recording (when `--record` is enabled)
F6	Pause/resume playback (when `--playback` is enabled)
F12	Take screenshot (saves to current directory)

Test Runner (designer-test.exe)

Automated test executor that:

Launches designer with testing options
Waits for window to appear
Reads UI hierarchy to find elements
Sends input events to simulate user interaction
Verifies results via log file
Outputs JSON test results

UI Hierarchy Format

The hierarchy JSON contains all visible UI elements:

{
  "timestamp": 1705312200,
  "screen": "apps/home/home.rml",
  "resolution": {"width": 677, "height": 1202},
  "elements": {
    "tag": "body",
    "id": "",
    "classes": ["home-screen"],
    "bounds": {"x": 0, "y": 0, "width": 677, "height": 1202},
    "visible": true,
    "text": null,
    "children": [
      {
        "tag": "div",
        "id": "dock-phone",
        "classes": ["dock-item"],
        "bounds": {"x": 85, "y": 1138, "width": 56, "height": 56},
        "visible": true,
        "text": null,
        "children": [...]
      }
    ]
  }
}

Key Fields

Field	Description
`tag`	HTML element tag (div, span, img, input)
`id`	Element ID attribute (empty if none)
`classes`	Array of CSS class names
`bounds`	Position and size in logical pixels
`visible`	Whether element is displayed
`text`	Text content (for leaf text nodes)
`children`	Nested child elements

Resolution Note

The resolution field contains RmlUi's logical dimensions, which may differ from physical window size due to DPI scaling. Tests must scale coordinates:

// Scale from hierarchy (logical) to window (physical)
int physicalX = logicalX * windowWidth / hierarchyWidth;
int physicalY = logicalY * windowHeight / hierarchyHeight;

Writing Tests

Test Function Signature

bool MyTest(TestContext& ctx);

TestContext provides:

ctx.window - WindowController for input
ctx.log - LogParser for log verification
ctx.hierarchy - HierarchyReader for element lookup

Finding Elements

// By ID
auto element = ctx.hierarchy.FindById("dock-phone");

// By class (returns all matches)
auto buttons = ctx.hierarchy.FindByClass("btn-icon");

// By tag
auto divs = ctx.hierarchy.FindByTag("div");

Clicking Elements

bool ClickById(TestContext& ctx, const std::string& id) {
    ctx.hierarchy.Reload();  // Get fresh hierarchy

    auto element = ctx.hierarchy.FindById(id);
    if (!element || !element->visible) {
        return false;
    }

    // Get center point
    int x = element->bounds.centerX();
    int y = element->bounds.centerY();

    // Scale to physical coordinates
    ScaleToPhysical(ctx, x, y);

    // Send click
    ctx.window.SendClick(x, y);
    return true;
}

bool TestNavigateToDialer(TestContext& ctx) {
    GoHome(ctx);       // Return to home screen
    ctx.log.Clear();   // Clear previous logs

    if (!ClickById(ctx, "dock-phone")) {
        return false;
    }

    // Wait for navigation animation
    std::this_thread::sleep_for(std::chrono::milliseconds(1000));

    // Check log for navigation event
    ctx.log.Reload();
    return ctx.log.Contains("Loaded screen: apps/dialer/dialer.rml");
}

// Go back to home screen by clicking back buttons
void GoHome(TestContext& ctx) {
    for (int i = 0; i < 5; ++i) {
        ctx.hierarchy.Reload();
        auto elements = ctx.hierarchy.FindByClass("app-bar-nav");
        if (!elements.empty() && elements[0].visible) {
            int x = elements[0].bounds.centerX();
            int y = elements[0].bounds.centerY();
            ScaleToPhysical(ctx, x, y);
            ctx.window.SendClick(x, y);
        }
        std::this_thread::sleep_for(std::chrono::milliseconds(400));
    }
    std::this_thread::sleep_for(std::chrono::milliseconds(800));
}

Registering Tests

In main.cpp:

TestRunner runner;
runner.SetDesignerPath(designerPath);
runner.SetDocumentPath(documentPath);
runner.SetLogPath(logPath);
runner.SetHierarchyPath(hierarchyPath);

// Register tests
runner.AddTest("Navigate to Dialer", TestNavigateToDialer);
runner.AddTest("Navigate to Messages", TestNavigateToMessages);
runner.AddTest("Navigate to Contacts", TestNavigateToContacts);
runner.AddTest("Navigate to Browser", TestNavigateToBrowser);
runner.AddTest("Navigation Sequence", TestNavigationSequence);

// Run all tests
auto results = runner.RunAll();
results.SaveToFile(resultsPath);

Test Results Format

{
  "name": "Mosis Designer UI Tests",
  "summary": {
    "passed": 5,
    "failed": 0,
    "skipped": 0,
    "errors": 0,
    "total": 5,
    "duration_ms": 31162.3
  },
  "tests": [
    {
      "name": "Navigate to Dialer",
      "status": "passed",
      "message": "Test passed",
      "duration_ms": 4216.88
    },
    {
      "name": "Navigate to Browser",
      "status": "passed",
      "message": "Test passed",
      "duration_ms": 4067.34
    }
  ]
}

Action Recording and Playback

The designer supports recording user interactions and playing them back for automated testing.

Recording Actions

# Start designer with recording enabled
mosis-designer.exe home.rml --record my-test.json

# Press F5 to start recording
# Interact with the UI (clicks, swipes, etc.)
# Press F5 again to stop and save

Recording is automatically saved when you close the window.

Playing Back Actions

# Play back a recorded test
mosis-designer.exe home.rml --playback my-test.json

Use F6 to pause/resume playback.

Action Recording Format

{
  "name": "Navigate to contacts",
  "description": "Test navigation flow",
  "screen_width": 540,
  "screen_height": 960,
  "initial_screen": "apps/home/home.rml",
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
  ]
}

Supported Action Types

Type	Fields	Description
`tap`	x, y, timestamp	Single tap at coordinates
`swipe`	x1, y1, x2, y2, duration, timestamp	Swipe gesture
`long_press`	x, y, duration, timestamp	Long press gesture
`button`	button, timestamp	Hardware button ("back", "home")
`wait`	duration, timestamp	Pause between actions
`key`	key_code, pressed, timestamp	Keyboard input

Creating Test Files Manually

You can also create test files manually using the UI hierarchy to find element coordinates:

# Get element coordinates from hierarchy
mosis-designer.exe home.rml --hierarchy hierarchy.json
# Read hierarchy.json to find element bounds
# Write action JSON with those coordinates

Screenshot Comparison

The testing framework includes pixel-level screenshot comparison for visual regression testing.

Using Screenshot Comparison

#include "testing/visual_capture.h"

// Capture a screenshot
mosis::testing::VisualCapture capture(540, 960);
capture.CaptureScreenshot("current.png");

// Compare two screenshots
float diff = mosis::testing::VisualCapture::CompareImages("baseline.png", "current.png");

// diff = 0.0 means identical
// diff = 1.0 means completely different
// Typical threshold: diff < 0.01 (less than 1% different)

Comparison Details

Compares RGBA pixels with a tolerance of 2 per channel
Returns ratio of differing pixels (0.0 to 1.0)
Different dimensions = 1.0 (completely different)
Missing files = 1.0 (comparison failed)

Visual Regression Test Example

bool TestVisualRegression(TestContext& ctx) {
    // Navigate to screen
    GoHome(ctx);
    ClickById(ctx, "dock-phone");
    std::this_thread::sleep_for(std::chrono::milliseconds(1000));

    // Capture screenshot
    mosis::testing::VisualCapture capture(ctx.width, ctx.height);
    capture.CaptureScreenshot("dialer-current.png");

    // Compare with baseline
    float diff = mosis::testing::VisualCapture::CompareImages(
        "baselines/dialer-expected.png",
        "dialer-current.png"
    );

    // Allow up to 1% difference
    return diff < 0.01f;
}

Running Tests

Command Line

# Run with defaults (auto-detects paths)
designer-test.exe

# Override paths
designer-test.exe --designer /path/to/mosis-designer.exe \
                  --document /path/to/home.rml \
                  --log /path/to/output.log \
                  --hierarchy /path/to/hierarchy.json \
                  --results /path/to/results.json

Exit Codes

0: All tests passed
1: One or more tests failed

Best Practices

Element Identification

Use IDs for clickable elements that tests need to find
Use semantic classes like app-bar-nav for navigation buttons
Avoid relying on position - use hierarchy lookup instead

Timing

Wait after navigation (1000ms) for animations to complete
Reload hierarchy before each element lookup
Use retry logic for file reads (hierarchy may be mid-write)

Test Independence

Start from known state - call GoHome() at start of each test
Clear logs before verification
Don't depend on previous test state

Adding Test IDs to RML

When creating new screens, add IDs to interactive elements:

<!-- Good: Has ID for testing -->
<div id="dock-phone" class="dock-item" onclick="navigateTo('dialer')">
    <img src="../../icons/phone.tga"/>
</div>

<!-- Good: Has class for back button detection -->
<div class="app-bar-nav btn-icon" onclick="goBack()">
    <img src="../../icons/back.tga"/>
</div>

Troubleshooting

Element Not Found

Verify the screen has loaded (check hierarchy screen field)
Check if element has the expected ID/class
Ensure element is visible (visible: true)

Click Not Registering

Check coordinate scaling (hierarchy vs window size)
Increase wait time for animations
Verify window is in foreground

Timing Issues

Increase delays after GoHome() and navigation
Add retries for hierarchy reload
Check for race conditions in file I/O

Android Testing

The Android app supports event injection for automated testing.

Event Injection Methods

The MainActivity provides these methods for programmatic testing:

// Inject touch events with normalized coordinates (0.0-1.0)
MainActivity.instance?.injectTouchDown(x, y)
MainActivity.instance?.injectTouchMove(x, y)
MainActivity.instance?.injectTouchUp(x, y)
MainActivity.instance?.injectClick(x, y)  // Down + delay + Up

Broadcast-Based Injection

For external test frameworks or ADB, send broadcasts:

# Inject a click at center (0.5, 0.5)
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
    --es touch_type "click" \
    --ef x 0.5 \
    --ef y 0.5

# Inject touch down
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
    --es touch_type "down" \
    --ef x 0.2 \
    --ef y 0.9

# Inject touch up
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
    --es touch_type "up" \
    --ef x 0.2 \
    --ef y 0.9

UI Element Coordinates

Dock items are positioned at the bottom of the screen. Approximate normalized positions:

Element	X	Y
dock-phone	0.16	0.97
dock-messages	0.39	0.97
dock-contacts	0.61	0.97
dock-browser	0.84	0.97

Instrumentation Tests

For full integration tests, use Android's instrumentation framework:

@RunWith(AndroidJUnit4::class)
class NavigationTest {
    @get:Rule
    val activityRule = ActivityScenarioRule(MainActivity::class.java)

    @Test
    fun testNavigateToDialer() {
        // Wait for service to connect
        Thread.sleep(2000)

        // Click on phone dock icon (normalized coords)
        activityRule.scenario.onActivity { activity ->
            activity.injectClick(0.16f, 0.97f)
        }

        // Wait for navigation
        Thread.sleep(1000)

        // Verify navigation occurred (check logs or UI state)
    }
}

Future Improvements

Action recording (capture user interactions) - CLI and infrastructure complete
Screenshot comparison (visual regression testing) - Pixel-level diff implemented
Action playback with timing - Fully functional
GLFW input hooks for automatic mouse recording - Complete via forked backend
Android instrumentation test suite
Parallel test execution
Test coverage reporting
Cross-platform test runner (desktop + Android)
Visual diff output (highlight changed pixels)

14 KiB Raw Blame History