move docs to docs/ folder, merge architecture files, update references
This commit is contained in:
534
docs/TESTING.md
Normal file
534
docs/TESTING.md
Normal file
@@ -0,0 +1,534 @@
|
||||
# Mosis UI Testing Framework
|
||||
|
||||
This document describes the automated UI testing framework for Mosis virtual smartphone.
|
||||
|
||||
## Overview
|
||||
|
||||
The testing framework enables automated validation of UI behavior through:
|
||||
|
||||
1. **UI Hierarchy Inspection**: JSON dump of all UI elements with bounds
|
||||
2. **Input Simulation**: Mouse clicks via Windows SendInput API
|
||||
3. **Log Verification**: Check navigation events via log file parsing
|
||||
4. **Test Results**: JSON output compatible with CI/CD pipelines
|
||||
5. **Action Recording/Playback**: Record and replay user interactions
|
||||
6. **Visual Regression**: Screenshot comparison with pixel-level diff
|
||||
|
||||
## Components
|
||||
|
||||
### Designer (mosis-designer.exe)
|
||||
|
||||
The desktop designer serves as the test target. When launched with testing options:
|
||||
|
||||
```bash
|
||||
mosis-designer.exe home.rml --log test.log --hierarchy hierarchy.json
|
||||
```
|
||||
|
||||
**Testing Options**:
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--log FILE` | Write all RmlUi INFO logs to file (navigation events, errors) |
|
||||
| `--hierarchy FILE` | Dump UI element tree to JSON each frame |
|
||||
| `--record FILE` | Enable action recording mode (F5 to start/stop) |
|
||||
| `--playback FILE` | Play back recorded actions from JSON file |
|
||||
|
||||
**Keyboard Controls**:
|
||||
|
||||
| Key | Function |
|
||||
|-----|----------|
|
||||
| F5 | Start/stop recording (when `--record` is enabled) |
|
||||
| F6 | Pause/resume playback (when `--playback` is enabled) |
|
||||
| F12 | Take screenshot (saves to current directory) |
|
||||
|
||||
### Test Runner (designer-test.exe)
|
||||
|
||||
Automated test executor that:
|
||||
|
||||
1. Launches designer with testing options
|
||||
2. Waits for window to appear
|
||||
3. Reads UI hierarchy to find elements
|
||||
4. Sends input events to simulate user interaction
|
||||
5. Verifies results via log file
|
||||
6. Outputs JSON test results
|
||||
|
||||
## UI Hierarchy Format
|
||||
|
||||
The hierarchy JSON contains all visible UI elements:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1705312200,
|
||||
"screen": "apps/home/home.rml",
|
||||
"resolution": {"width": 677, "height": 1202},
|
||||
"elements": {
|
||||
"tag": "body",
|
||||
"id": "",
|
||||
"classes": ["home-screen"],
|
||||
"bounds": {"x": 0, "y": 0, "width": 677, "height": 1202},
|
||||
"visible": true,
|
||||
"text": null,
|
||||
"children": [
|
||||
{
|
||||
"tag": "div",
|
||||
"id": "dock-phone",
|
||||
"classes": ["dock-item"],
|
||||
"bounds": {"x": 85, "y": 1138, "width": 56, "height": 56},
|
||||
"visible": true,
|
||||
"text": null,
|
||||
"children": [...]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `tag` | HTML element tag (div, span, img, input) |
|
||||
| `id` | Element ID attribute (empty if none) |
|
||||
| `classes` | Array of CSS class names |
|
||||
| `bounds` | Position and size in logical pixels |
|
||||
| `visible` | Whether element is displayed |
|
||||
| `text` | Text content (for leaf text nodes) |
|
||||
| `children` | Nested child elements |
|
||||
|
||||
### Resolution Note
|
||||
|
||||
The `resolution` field contains RmlUi's logical dimensions, which may differ from physical window size due to DPI scaling. Tests must scale coordinates:
|
||||
|
||||
```cpp
|
||||
// Scale from hierarchy (logical) to window (physical)
|
||||
int physicalX = logicalX * windowWidth / hierarchyWidth;
|
||||
int physicalY = logicalY * windowHeight / hierarchyHeight;
|
||||
```
|
||||
|
||||
## Writing Tests
|
||||
|
||||
### Test Function Signature
|
||||
|
||||
```cpp
|
||||
bool MyTest(TestContext& ctx);
|
||||
```
|
||||
|
||||
TestContext provides:
|
||||
- `ctx.window` - WindowController for input
|
||||
- `ctx.log` - LogParser for log verification
|
||||
- `ctx.hierarchy` - HierarchyReader for element lookup
|
||||
|
||||
### Finding Elements
|
||||
|
||||
```cpp
|
||||
// By ID
|
||||
auto element = ctx.hierarchy.FindById("dock-phone");
|
||||
|
||||
// By class (returns all matches)
|
||||
auto buttons = ctx.hierarchy.FindByClass("btn-icon");
|
||||
|
||||
// By tag
|
||||
auto divs = ctx.hierarchy.FindByTag("div");
|
||||
```
|
||||
|
||||
### Clicking Elements
|
||||
|
||||
```cpp
|
||||
bool ClickById(TestContext& ctx, const std::string& id) {
|
||||
ctx.hierarchy.Reload(); // Get fresh hierarchy
|
||||
|
||||
auto element = ctx.hierarchy.FindById(id);
|
||||
if (!element || !element->visible) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Get center point
|
||||
int x = element->bounds.centerX();
|
||||
int y = element->bounds.centerY();
|
||||
|
||||
// Scale to physical coordinates
|
||||
ScaleToPhysical(ctx, x, y);
|
||||
|
||||
// Send click
|
||||
ctx.window.SendClick(x, y);
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
### Verifying Navigation
|
||||
|
||||
```cpp
|
||||
bool TestNavigateToDialer(TestContext& ctx) {
|
||||
GoHome(ctx); // Return to home screen
|
||||
ctx.log.Clear(); // Clear previous logs
|
||||
|
||||
if (!ClickById(ctx, "dock-phone")) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Wait for navigation animation
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
|
||||
|
||||
// Check log for navigation event
|
||||
ctx.log.Reload();
|
||||
return ctx.log.Contains("Loaded screen: apps/dialer/dialer.rml");
|
||||
}
|
||||
```
|
||||
|
||||
### Navigation Helpers
|
||||
|
||||
```cpp
|
||||
// Go back to home screen by clicking back buttons
|
||||
void GoHome(TestContext& ctx) {
|
||||
for (int i = 0; i < 5; ++i) {
|
||||
ctx.hierarchy.Reload();
|
||||
auto elements = ctx.hierarchy.FindByClass("app-bar-nav");
|
||||
if (!elements.empty() && elements[0].visible) {
|
||||
int x = elements[0].bounds.centerX();
|
||||
int y = elements[0].bounds.centerY();
|
||||
ScaleToPhysical(ctx, x, y);
|
||||
ctx.window.SendClick(x, y);
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(400));
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(800));
|
||||
}
|
||||
```
|
||||
|
||||
## Registering Tests
|
||||
|
||||
In `main.cpp`:
|
||||
|
||||
```cpp
|
||||
TestRunner runner;
|
||||
runner.SetDesignerPath(designerPath);
|
||||
runner.SetDocumentPath(documentPath);
|
||||
runner.SetLogPath(logPath);
|
||||
runner.SetHierarchyPath(hierarchyPath);
|
||||
|
||||
// Register tests
|
||||
runner.AddTest("Navigate to Dialer", TestNavigateToDialer);
|
||||
runner.AddTest("Navigate to Messages", TestNavigateToMessages);
|
||||
runner.AddTest("Navigate to Contacts", TestNavigateToContacts);
|
||||
runner.AddTest("Navigate to Browser", TestNavigateToBrowser);
|
||||
runner.AddTest("Navigation Sequence", TestNavigationSequence);
|
||||
|
||||
// Run all tests
|
||||
auto results = runner.RunAll();
|
||||
results.SaveToFile(resultsPath);
|
||||
```
|
||||
|
||||
## Test Results Format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Mosis Designer UI Tests",
|
||||
"summary": {
|
||||
"passed": 5,
|
||||
"failed": 0,
|
||||
"skipped": 0,
|
||||
"errors": 0,
|
||||
"total": 5,
|
||||
"duration_ms": 31162.3
|
||||
},
|
||||
"tests": [
|
||||
{
|
||||
"name": "Navigate to Dialer",
|
||||
"status": "passed",
|
||||
"message": "Test passed",
|
||||
"duration_ms": 4216.88
|
||||
},
|
||||
{
|
||||
"name": "Navigate to Browser",
|
||||
"status": "passed",
|
||||
"message": "Test passed",
|
||||
"duration_ms": 4067.34
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Action Recording and Playback
|
||||
|
||||
The designer supports recording user interactions and playing them back for automated testing.
|
||||
|
||||
### Recording Actions
|
||||
|
||||
```bash
|
||||
# Start designer with recording enabled
|
||||
mosis-designer.exe home.rml --record my-test.json
|
||||
|
||||
# Press F5 to start recording
|
||||
# Interact with the UI (clicks, swipes, etc.)
|
||||
# Press F5 again to stop and save
|
||||
```
|
||||
|
||||
Recording is automatically saved when you close the window.
|
||||
|
||||
### Playing Back Actions
|
||||
|
||||
```bash
|
||||
# Play back a recorded test
|
||||
mosis-designer.exe home.rml --playback my-test.json
|
||||
```
|
||||
|
||||
Use F6 to pause/resume playback.
|
||||
|
||||
### Action Recording Format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Navigate to contacts",
|
||||
"description": "Test navigation flow",
|
||||
"screen_width": 540,
|
||||
"screen_height": 960,
|
||||
"initial_screen": "apps/home/home.rml",
|
||||
"actions": [
|
||||
{"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
|
||||
{"type": "wait", "duration": 1000, "timestamp": 100},
|
||||
{"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
|
||||
{"type": "swipe", "x1": 100, "y1": 500, "x2": 100, "y2": 200, "duration": 300, "timestamp": 2000}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Supported Action Types
|
||||
|
||||
| Type | Fields | Description |
|
||||
|------|--------|-------------|
|
||||
| `tap` | x, y, timestamp | Single tap at coordinates |
|
||||
| `swipe` | x1, y1, x2, y2, duration, timestamp | Swipe gesture |
|
||||
| `long_press` | x, y, duration, timestamp | Long press gesture |
|
||||
| `button` | button, timestamp | Hardware button ("back", "home") |
|
||||
| `wait` | duration, timestamp | Pause between actions |
|
||||
| `key` | key_code, pressed, timestamp | Keyboard input |
|
||||
|
||||
### Creating Test Files Manually
|
||||
|
||||
You can also create test files manually using the UI hierarchy to find element coordinates:
|
||||
|
||||
```bash
|
||||
# Get element coordinates from hierarchy
|
||||
mosis-designer.exe home.rml --hierarchy hierarchy.json
|
||||
# Read hierarchy.json to find element bounds
|
||||
# Write action JSON with those coordinates
|
||||
```
|
||||
|
||||
## Screenshot Comparison
|
||||
|
||||
The testing framework includes pixel-level screenshot comparison for visual regression testing.
|
||||
|
||||
### Using Screenshot Comparison
|
||||
|
||||
```cpp
|
||||
#include "testing/visual_capture.h"
|
||||
|
||||
// Capture a screenshot
|
||||
mosis::testing::VisualCapture capture(540, 960);
|
||||
capture.CaptureScreenshot("current.png");
|
||||
|
||||
// Compare two screenshots
|
||||
float diff = mosis::testing::VisualCapture::CompareImages("baseline.png", "current.png");
|
||||
|
||||
// diff = 0.0 means identical
|
||||
// diff = 1.0 means completely different
|
||||
// Typical threshold: diff < 0.01 (less than 1% different)
|
||||
```
|
||||
|
||||
### Comparison Details
|
||||
|
||||
- Compares RGBA pixels with a tolerance of 2 per channel
|
||||
- Returns ratio of differing pixels (0.0 to 1.0)
|
||||
- Different dimensions = 1.0 (completely different)
|
||||
- Missing files = 1.0 (comparison failed)
|
||||
|
||||
### Visual Regression Test Example
|
||||
|
||||
```cpp
|
||||
bool TestVisualRegression(TestContext& ctx) {
|
||||
// Navigate to screen
|
||||
GoHome(ctx);
|
||||
ClickById(ctx, "dock-phone");
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
|
||||
|
||||
// Capture screenshot
|
||||
mosis::testing::VisualCapture capture(ctx.width, ctx.height);
|
||||
capture.CaptureScreenshot("dialer-current.png");
|
||||
|
||||
// Compare with baseline
|
||||
float diff = mosis::testing::VisualCapture::CompareImages(
|
||||
"baselines/dialer-expected.png",
|
||||
"dialer-current.png"
|
||||
);
|
||||
|
||||
// Allow up to 1% difference
|
||||
return diff < 0.01f;
|
||||
}
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Command Line
|
||||
|
||||
```bash
|
||||
# Run with defaults (auto-detects paths)
|
||||
designer-test.exe
|
||||
|
||||
# Override paths
|
||||
designer-test.exe --designer /path/to/mosis-designer.exe \
|
||||
--document /path/to/home.rml \
|
||||
--log /path/to/output.log \
|
||||
--hierarchy /path/to/hierarchy.json \
|
||||
--results /path/to/results.json
|
||||
```
|
||||
|
||||
### Exit Codes
|
||||
|
||||
- `0`: All tests passed
|
||||
- `1`: One or more tests failed
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Element Identification
|
||||
|
||||
1. **Use IDs** for clickable elements that tests need to find
|
||||
2. **Use semantic classes** like `app-bar-nav` for navigation buttons
|
||||
3. **Avoid relying on position** - use hierarchy lookup instead
|
||||
|
||||
### Timing
|
||||
|
||||
1. **Wait after navigation** (1000ms) for animations to complete
|
||||
2. **Reload hierarchy** before each element lookup
|
||||
3. **Use retry logic** for file reads (hierarchy may be mid-write)
|
||||
|
||||
### Test Independence
|
||||
|
||||
1. **Start from known state** - call GoHome() at start of each test
|
||||
2. **Clear logs** before verification
|
||||
3. **Don't depend on previous test state**
|
||||
|
||||
## Adding Test IDs to RML
|
||||
|
||||
When creating new screens, add IDs to interactive elements:
|
||||
|
||||
```html
|
||||
<!-- Good: Has ID for testing -->
|
||||
<div id="dock-phone" class="dock-item" onclick="navigateTo('dialer')">
|
||||
<img src="../../icons/phone.tga"/>
|
||||
</div>
|
||||
|
||||
<!-- Good: Has class for back button detection -->
|
||||
<div class="app-bar-nav btn-icon" onclick="goBack()">
|
||||
<img src="../../icons/back.tga"/>
|
||||
</div>
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Element Not Found
|
||||
|
||||
- Verify the screen has loaded (check hierarchy `screen` field)
|
||||
- Check if element has the expected ID/class
|
||||
- Ensure element is visible (`visible: true`)
|
||||
|
||||
### Click Not Registering
|
||||
|
||||
- Check coordinate scaling (hierarchy vs window size)
|
||||
- Increase wait time for animations
|
||||
- Verify window is in foreground
|
||||
|
||||
### Timing Issues
|
||||
|
||||
- Increase delays after GoHome() and navigation
|
||||
- Add retries for hierarchy reload
|
||||
- Check for race conditions in file I/O
|
||||
|
||||
## Android Testing
|
||||
|
||||
The Android app supports event injection for automated testing.
|
||||
|
||||
### Event Injection Methods
|
||||
|
||||
The `MainActivity` provides these methods for programmatic testing:
|
||||
|
||||
```kotlin
|
||||
// Inject touch events with normalized coordinates (0.0-1.0)
|
||||
MainActivity.instance?.injectTouchDown(x, y)
|
||||
MainActivity.instance?.injectTouchMove(x, y)
|
||||
MainActivity.instance?.injectTouchUp(x, y)
|
||||
MainActivity.instance?.injectClick(x, y) // Down + delay + Up
|
||||
```
|
||||
|
||||
### Broadcast-Based Injection
|
||||
|
||||
For external test frameworks or ADB, send broadcasts:
|
||||
|
||||
```bash
|
||||
# Inject a click at center (0.5, 0.5)
|
||||
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
||||
--es touch_type "click" \
|
||||
--ef x 0.5 \
|
||||
--ef y 0.5
|
||||
|
||||
# Inject touch down
|
||||
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
||||
--es touch_type "down" \
|
||||
--ef x 0.2 \
|
||||
--ef y 0.9
|
||||
|
||||
# Inject touch up
|
||||
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
||||
--es touch_type "up" \
|
||||
--ef x 0.2 \
|
||||
--ef y 0.9
|
||||
```
|
||||
|
||||
### UI Element Coordinates
|
||||
|
||||
Dock items are positioned at the bottom of the screen. Approximate normalized positions:
|
||||
|
||||
| Element | X | Y |
|
||||
|---------|---|---|
|
||||
| dock-phone | 0.16 | 0.97 |
|
||||
| dock-messages | 0.39 | 0.97 |
|
||||
| dock-contacts | 0.61 | 0.97 |
|
||||
| dock-browser | 0.84 | 0.97 |
|
||||
|
||||
### Instrumentation Tests
|
||||
|
||||
For full integration tests, use Android's instrumentation framework:
|
||||
|
||||
```kotlin
|
||||
@RunWith(AndroidJUnit4::class)
|
||||
class NavigationTest {
|
||||
@get:Rule
|
||||
val activityRule = ActivityScenarioRule(MainActivity::class.java)
|
||||
|
||||
@Test
|
||||
fun testNavigateToDialer() {
|
||||
// Wait for service to connect
|
||||
Thread.sleep(2000)
|
||||
|
||||
// Click on phone dock icon (normalized coords)
|
||||
activityRule.scenario.onActivity { activity ->
|
||||
activity.injectClick(0.16f, 0.97f)
|
||||
}
|
||||
|
||||
// Wait for navigation
|
||||
Thread.sleep(1000)
|
||||
|
||||
// Verify navigation occurred (check logs or UI state)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Future Improvements
|
||||
|
||||
- [x] Action recording (capture user interactions) - *CLI and infrastructure complete*
|
||||
- [x] Screenshot comparison (visual regression testing) - *Pixel-level diff implemented*
|
||||
- [x] Action playback with timing - *Fully functional*
|
||||
- [x] GLFW input hooks for automatic mouse recording - *Complete via forked backend*
|
||||
- [ ] Android instrumentation test suite
|
||||
- [ ] Parallel test execution
|
||||
- [ ] Test coverage reporting
|
||||
- [ ] Cross-platform test runner (desktop + Android)
|
||||
- [ ] Visual diff output (highlight changed pixels)
|
||||
Reference in New Issue
Block a user