# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Company **OmixLab LTD** - Package namespace: `com.omixlab` ## Git Commit Guidelines **IMPORTANT**: When creating git commits: - **DO NOT** add yourself as a co-author (no `Co-Authored-By` lines) - **Commit messages must be a single line** - keep it concise and descriptive ## Project Overview Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users can interact with inside VR environments, with real smartphone functionality. ### Current Status | Component | Status | Notes | |-----------|--------|-------| | MosisService | ✅ Working | RmlUi rendering, touch input, navigation | | App Management | ✅ Working | Install/uninstall apps, sandbox integration | | Lua Sandbox | ✅ Working | 149 security tests passing | | Desktop Designer | ✅ Working | Hot-reload, hierarchy dump, recording | | Designer Tests | ✅ 5/5 Passing | Navigation tests automated | | MosisVR (Unity) | ✅ Building | OpenGL backend working, Vulkan in progress | | MosisUnreal | ✅ Working | Vulkan texture import via UE5 RHI, phone actor with mesh | ### Project Components | Component | Location | Purpose | |-----------|----------|---------| | Android Service | `src/main/` | Native service running RmlUi renderer | | App Management | `src/main/cpp/apps/` | App install/uninstall/launch with sandbox | | Lua Sandbox | `src/main/cpp/sandbox/` | Per-app Lua isolation (22 modules) | | Desktop Designer | `designer/` | UI development with hot-reload | | Designer Tests | `designer-test/` | Automated UI testing framework | | Sandbox Tests | `sandbox-test/` | Lua sandbox security tests (149 tests) | | UI Assets | `src/main/assets/` | Shared RML/RCSS/Lua assets | ## Build Commands ### Android (Gradle) ```bash # Build entire project ./gradlew build # Build debug APK ./gradlew assembleDebug # Build release APK ./gradlew assembleRelease # Clean build outputs ./gradlew clean # Build with verbose output ./gradlew build --info --stacktrace # Run lint checks ./gradlew lint # Run unit tests ./gradlew test # Run connected device tests ./gradlew connectedAndroidTest ``` ### Desktop Designer (CMake) ```bash # Configure (from designer/ folder) cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake # Build cmake --build build --config Debug # Run with hot-reload ./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml # Run with logging and hierarchy dump ./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml --log output.log --hierarchy hierarchy.json ``` ### Designer Tests (CMake) ```bash # Configure (from designer-test/ folder) cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake # Build cmake --build build --config Debug # Run tests ./build/Debug/designer-test.exe ``` ### Sandbox Security Tests (CMake) ```bash # Configure (from sandbox-test/ folder) cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake # Build cmake --build build --config Debug # Run all tests (uber command) ./run_tests.bat # Or run directly ./build/Debug/sandbox-test.exe # Run specific test ./build/Debug/sandbox-test.exe --test DangerousGlobals ./build/Debug/sandbox-test.exe --test Memory ./build/Debug/sandbox-test.exe --test CPU ``` ## Environment Requirements Required environment variables: - `ANDROID_HOME` - Android SDK path - `ANDROID_NDK_HOME` - Android NDK path (version 29.0.14206865) - `VCPKG_ROOT` - vcpkg package manager root ## Architecture Overview MosisService is an Android application combining Kotlin UI with native C++ libraries for UI rendering via Android's Binder IPC system. ### Two Native Libraries **mosis-service** (`libmosis-service.so`): - Main Android Binder service implementation - Implements `IMosisService.aidl` interface for touch events and initialization - Contains the Kernel rendering engine with RmlUi integration - Links against RmlUi for HTML/CSS-like UI rendering **mosis-test** (`libmosis-test.so`): - Test/rendering client implementation - Implements `IMosisListener.aidl` for receiving callbacks - OpenGL ES 2.0 rendering pipeline using GLAD ### IPC Flow ``` Kotlin NativeService → JNI → mosis-service (IMosisService) ↓ IMosisListener callbacks ↓ mosis-test (rendering client) ``` ### Key Interfaces (AIDL) `IMosisService`: `initOS()`, `onTouchDown()`, `onTouchMove()`, `onTouchUp()` `IMosisListener` (oneway/async): `onServiceInitialized()`, `onBufferAvailable()`, `onFrameAvailable()` ### Native Code Structure (src/main/cpp/) **Core:** - `kernel.cpp` - Core rendering engine, RmlUi integration, event processing - `mosis-service.cpp` - Binder service implementation, JNI entry points - `mosis-test.cpp` - Test client implementation - `egl_context.cpp` - OpenGL ES context management - `render_target.cpp` - Framebuffer and buffer management - `RmlUi_Renderer_GL3.cpp` - RmlUi OpenGL renderer backend - `assets_manager.cpp` - Android AssetManager integration **App Management (`apps/`):** - `app_manager.cpp` - App install/uninstall/launch lifecycle - `app_api.cpp` - Lua API bindings for app management - `update_service.cpp` - Background update checking **Lua Sandbox (`sandbox/`):** - `sandbox_manager.cpp` - Multi-app sandbox orchestrator - `lua_sandbox.cpp` - Core Lua sandbox with resource limits - `permission_gate.cpp` - Permission system (normal/dangerous/signature) - `virtual_fs.cpp` - Per-app virtual filesystem with quotas - `database_manager.cpp` - SQLite database per app - `network_manager.cpp` - HTTP request validation - `websocket_manager.cpp` - WebSocket connections - `timer_manager.cpp` - setTimeout/setInterval implementation - `json_api.cpp` - Safe JSON encode/decode - `crypto_api.cpp` - Cryptographic functions (SHA256, HMAC) - `camera_interface.cpp` - Camera access with indicators - `microphone_interface.cpp` - Microphone access with indicators - `audio_output.cpp` - Audio playback - `location_interface.cpp` - GPS with precision reduction - `sensor_interface.cpp` - Accelerometer, gyroscope, etc. - `bluetooth_interface.cpp` - Bluetooth device access - `contacts_interface.cpp` - Contacts read/write - `message_bus.cpp` - Inter-app communication ## Code Style - C++23 standard with modern features (std::span, std::format) - PascalCase for classes/functions, camelCase for variables - RAII principles with smart pointers - Kotlin code follows Android conventions ## Dependencies - vcpkg manages native dependencies (RmlUi, GLFW, Freetype, Lua, libpng, nlohmann-json, minizip, sqlite3) - CMake build system with vcpkg toolchain integration - Android target architecture: arm64-v8a only - Desktop target: Windows x64 (MSVC) ## Desktop Designer The desktop designer (`designer/`) provides rapid UI development with: - **Hot-reload**: Automatically reloads when RML/RCSS/Lua files change - **UI Hierarchy Dumping**: Exports element tree to JSON for inspection - **Screenshot Capture**: PNG export via F12 key - **Logging**: Detailed output for debugging navigation and events - **Action Recording**: Record mouse/keyboard interactions to JSON - **Action Playback**: Replay recorded interactions with timing ### Key Files | File | Purpose | |------|---------| | `designer/src/main.cpp` | Main entry point, GLFW window, event loop | | `designer/src/desktop_kernel.cpp` | RmlUi context management, rendering | | `designer/src/testing/ui_inspector.cpp` | UI hierarchy JSON export | | `designer/src/testing/visual_capture.cpp` | PNG screenshot capture and comparison | | `designer/src/testing/action_recorder.cpp` | Record user interactions to JSON | | `designer/src/testing/action_player.cpp` | Playback recorded actions | | `designer/src/backend/RmlUi_Backend_GLFW_GL3.cpp` | GLFW backend with input hooks | ### Command Line Options ``` --log Write logs to file --hierarchy Dump UI hierarchy JSON each frame --dump Single-shot dump mode (screenshot + hierarchy) --record Enable recording mode (F5 to start/stop) --playback Play back recorded actions from JSON ``` ### Keyboard Controls | Key | Function | |-----|----------| | F5 | Start/stop recording (when --record is enabled) | | F6 | Pause/resume playback (when --playback is enabled) | | F12 | Take screenshot | ## Automated Testing Framework The designer-test (`designer-test/`) provides automated UI testing: ### Test Architecture 1. **WindowController**: Finds designer window, sends mouse/keyboard events via Windows SendInput API 2. **HierarchyReader**: Parses UI hierarchy JSON to find elements by ID/class (with retry logic and exponential backoff) 3. **LogParser**: Monitors log file for navigation events 4. **TestRunner**: Orchestrates test execution, reports results 5. **UIInspector**: Dumps UI hierarchy with atomic writes (temp file + rename pattern) ### Key Implementation Details - **Path Normalization**: RmlUi uses `|` instead of `:` in Windows paths (e.g., `D|\Dev\...`). The UIInspector normalizes paths for correct document matching. - **Atomic File Writes**: Hierarchy files are written to `.tmp` then renamed to prevent partial reads. - **Retry with Backoff**: HierarchyReader retries up to 10 times with exponential backoff (30ms base) and validates JSON completeness. - **Dynamic Back Button**: `GoHome()` finds back buttons from hierarchy by class (`app-bar-nav` or `browser-nav-btn`) instead of fixed coordinates. ### Writing Tests ```cpp // Find element by ID and click it bool ClickById(TestContext& ctx, const std::string& id) { ctx.hierarchy.Reload(); auto element = ctx.hierarchy.FindById(id); if (!element) return false; int x = element->bounds.centerX(); int y = element->bounds.centerY(); ScaleToPhysical(ctx, x, y); // Convert logical to physical coords ctx.window.SendClick(x, y); return true; } // Test example bool TestNavigateToDialer(TestContext& ctx) { GoHome(ctx); ctx.log.Clear(); if (!ClickById(ctx, "dock-phone")) return false; std::this_thread::sleep_for(std::chrono::milliseconds(1000)); ctx.log.Reload(); return ctx.log.Contains("Loaded screen: apps/dialer/dialer.rml"); } ``` ### Test Output Tests produce JSON results at `test_results.json`: ```json { "name": "Mosis Designer UI Tests", "summary": {"passed": 5, "failed": 0, "total": 5}, "tests": [ {"name": "Navigate to Dialer", "status": "passed", "duration_ms": 3500} ] } ``` ## UI Assets Structure All UI assets are in `src/main/assets/`: ``` assets/ ├── apps/ # System apps (RML documents) │ ├── home/home.rml # Home screen launcher │ ├── dialer/dialer.rml # Phone dialer │ ├── messages/ # Messages app │ ├── contacts/ # Contacts app │ ├── settings/ # Settings app │ └── browser/ # Web browser ├── ui/ # Shared stylesheets │ ├── html.rcss # Base HTML element styles │ ├── theme.rcss # Design tokens (colors, typography) │ └── components.rcss # Reusable UI components ├── scripts/ # Lua scripts │ └── navigation.lua # Navigation system ├── icons/ # TGA icon files (24x24, 32x32) └── fonts/ # TTF fonts (LatoLatin, Roboto) ``` ### Navigation System Navigation is handled by `scripts/navigation.lua`: ```lua -- Navigate to a screen navigateTo('dialer') -- Push to history, animate forward -- Go back goBack() -- Pop from history, animate back -- Go home goHome() -- Clear history, return to home ``` ### Element IDs for Testing Key elements have IDs for automated testing: | ID | Location | Purpose | |----|----------|---------| | `dock-phone` | home.rml | Phone dock icon | | `dock-messages` | home.rml | Messages dock icon | | `dock-contacts` | home.rml | Contacts dock icon | | `dock-browser` | home.rml | Browser dock icon | | `app-settings` | home.rml | Settings app icon | ### CSS Classes for Navigation Back buttons use `app-bar-nav` class for automated GoHome: ```html
``` Browser uses `browser-nav-btn` class for its toolbar back button: ```html
``` The test framework's `FindBackButton()` searches for both classes to handle all screen layouts. ## Material Design Resources Material Design icons and components are available in the MosisDesigner repository: ### Material Design Icons **Location**: `D:\Dev\Mosis\MosisDesigner\material-design-icons` A comprehensive icon library from Google with 2000+ icons across 20 categories: | Category | Examples | |----------|----------| | action | home, search, settings, delete, info | | alert | error, warning, notification | | av | play, pause, volume, mic | | communication | phone, message, email, contacts | | content | add, remove, copy, paste | | device | battery, wifi, bluetooth, gps | | editor | format, text, color, brush | | file | folder, attachment, download, upload | | hardware | keyboard, mouse, phone, tablet | | home | lightbulb, thermostat, security | | image | camera, photo, filter, tune | | maps | location, directions, navigation | | navigation | arrow, chevron, menu, close | | notification | sync, update, event | | places | hotel, restaurant, airport | | search | search variants | | social | share, person, group, notifications | | toggle | star, checkbox, radio | **Available Formats**: - `src/` - SVG source files organized by category - `png/` - PNG files at multiple DPIs (24dp, 36dp, 48dp) - `font/` - Icon fonts (WOFF, TTF) - `symbols/` - Material Symbols variable font (newer) - `variablefont/` - Variable font files **Icon Styles**: - Outlined (default) - Filled - Rounded - Sharp - Two-tone (Material Icons only) ### Material Design Lite **Location**: `D:\Dev\Mosis\MosisDesigner\material-design-lite` CSS/JS component library implementing Material Design (reference implementation): | Directory | Contents | |-----------|----------| | `src/` | SASS source for components | | `docs/` | Component documentation | | `templates/` | Page templates | **Key Components** (for design reference): - Buttons (raised, flat, FAB) - Cards - Dialogs - Lists - Menus - Navigation drawers - Progress indicators - Sliders - Snackbars - Tables - Tabs - Text fields - Tooltips ### Using Icons in Mosis 1. **Find icon** at https://fonts.google.com/icons 2. **Export SVG** from `material-design-icons/src///` 3. **Convert to TGA** using image tool (24x24 or 32x32, RGBA) 4. **Place in** `src/main/assets/icons/` 5. **Reference in RML**: `` ## Android Device Testing ### Prerequisites ```bash # Check connected device adb devices -l # Verify Mosis app is installed adb shell pm list packages | grep mosis ``` ### Build and Install ```bash # Build debug APK ./gradlew assembleDebug # Install on device adb install -r build/outputs/apk/debug/MosisService-debug.apk # Launch the app adb shell am start -n com.omixlab.mosis/.MainActivity ``` ### Run Gradle Connected Tests ```bash # Run all connected Android tests ./gradlew connectedAndroidTest ``` ### Event Injection via ADB Inject touch events for automated testing: ```bash # Click at normalized coordinates (0.0-1.0) adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.5 --ef y 0.5 # Touch down adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "down" --ef x 0.2 --ef y 0.9 # Touch up adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "up" --ef x 0.2 --ef y 0.9 ``` ### Dock Element Coordinates (Normalized) | Element | X | Y | |---------|---|---| | dock-phone | 0.16 | 0.97 | | dock-messages | 0.39 | 0.97 | | dock-contacts | 0.61 | 0.97 | | dock-browser | 0.84 | 0.97 | | back-button | 0.10 | 0.05 | ### Full Navigation Test Sequence ```bash # Clear logs and run navigation test sequence adb logcat -c # Click Phone dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.16 --ef y 0.97 sleep 2 # Click back to return home adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.1 --ef y 0.05 sleep 2 # Click Messages dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.39 --ef y 0.97 sleep 2 # Click back to return home adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.1 --ef y 0.05 sleep 2 # Click Contacts dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.61 --ef y 0.97 sleep 2 # Click back to return home adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.1 --ef y 0.05 sleep 2 # Click Browser dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.84 --ef y 0.97 ``` ### Reading Logs ```bash # Filter for Mosis logs adb logcat -s MosisTest ServiceTester RMLUI # Filter for navigation events adb logcat -d | grep -iE "navigat|loaded|goBack|rml" # Save to file adb logcat -s MosisTest > mosis-log.txt # Clear logs adb logcat -c ``` ### Expected Log Output Successful navigation shows these log patterns: ``` RMLUI: navigateTo called with: dialer Loading screen: apps/dialer/dialer.rml RMLUI: Navigated to: dialer (history depth: 1) RMLUI: goBack called (history depth: 1) Loading screen: apps/home/home.rml RMLUI: Back to: home ``` ## Documentation Guidelines **IMPORTANT**: Always document progress and new commands to avoid rediscovery. ### Where to Put Documentation | Content Type | Location | |--------------|----------| | General concepts, architecture | `MosisService/CLAUDE.md` (this file) | | Unreal plugin docs | `MosisUnreal/Plugins/MosisSDK/README.md` | | Unity package docs | `MosisVR/Packages/com.omixlab.mosis_sdk/README.md` | | Project-specific build commands | Each project's own docs | **DO NOT** put documentation in the root `D:\Dev\Mosis\` directory - it is not versioned. ### What to Document 1. **Build commands** - Every new build command discovered or created 2. **Environment setup** - Required environment variables, SDK versions 3. **Architecture decisions** - Why something was done a certain way 4. **Issues and solutions** - Problems encountered and how they were fixed 5. **File locations** - Where important files are and what they do ### When to Document - After completing a milestone or feature - When discovering new build commands - When fixing a non-obvious issue - When adding new dependencies or requirements ## Game Engine Integrations MosisService provides a virtual phone that game engines can display and interact with. ### Integration Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ MosisService │ │ (OpenGL ES rendering → AHardwareBuffer) │ └───────────────────────────┬─────────────────────────────────────┘ │ Binder IPC + Shared Memory ┌─────────────────┴─────────────────┐ ▼ ▼ ┌─────────────────────┐ ┌─────────────────────┐ │ MosisUnreal │ │ MosisVR │ │ (UE5.5 Plugin) │ │ (Unity Package) │ │ Vulkan Import │ │ Vulkan/OpenGL │ └─────────────────────┘ └─────────────────────┘ ``` ### MosisUnreal (Unreal Engine 5.5) **Location**: `D:\Dev\Mosis\MosisUnreal\Plugins\MosisSDK\` #### Build Commands ```batch :: Windows Editor Build "D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^ MosisUnrealEditor Win64 Development ^ -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" :: Android APK Build "D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^ BuildCookRun ^ -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^ -platform=Android -clientconfig=Development ^ -build -cook -stage -pak -package -noP4 :: Clean Build (delete these folders first) rmdir /s /q "Intermediate\Build" rmdir /s /q "Binaries" ``` #### Output Files | Build | Output | |-------|--------| | Windows Editor | `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll` | | Android APK | `Binaries/Android/MosisUnreal-arm64.apk` | | Android OBB | `Binaries/Android/main.1.com.omixlab.MosisUnreal.obb` | #### Requirements - Android SDK Platform 36 (for AIDL binder headers) - Android Build Tools 36.1.0 (for AIDL compiler) - `ANDROID_HOME` environment variable set #### Quest Deployment **IMPORTANT**: Git Bash on Windows converts Unix paths to Windows paths. Use `MSYS_NO_PATHCONV=1` prefix for all adb push commands. ```bash # Set Quest device ID (check with: adb devices -l) QUEST=2G0YC5ZF7W01T0 # Install APK adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk # Create OBB temp directory adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal" # Push OBB to temp location (MUST use MSYS_NO_PATHCONV=1) MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/ # Move OBB to final location adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal" adb -s $QUEST shell "mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/" # Start MosisService first, then MosisUnreal adb -s $QUEST shell am start -n com.omixlab.mosis/.MainActivity sleep 3 adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity # Monitor logs adb -s $QUEST logcat -s MosisSDK MosisOS MosisTest ``` **Full rebuild and deploy sequence**: ```bash # 1. Build Android APK+OBB "D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun \ -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" \ -platform=Android -clientconfig=Development \ -build -cook -stage -pak -package -noP4 # 2. Stop app, push APK and OBB, restart QUEST=2G0YC5ZF7W01T0 adb -s $QUEST shell am force-stop com.omixlab.MosisUnreal adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal" MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/ adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal && mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/" adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity ``` **Common issues**: - App stuck on loading screen: OBB not pushed or wrong version - `adb push` path errors: Missing `MSYS_NO_PATHCONV=1` prefix - Service not connecting: Start `com.omixlab.mosis` before the game #### VR Pointer Interaction The MosisSDK includes `UMosisPointerComponent` for VR ray-based touch interaction. **Key Files**: - `Public/MosisPointerComponent.h` - Component header - `Private/MosisPointerComponent.cpp` - Implementation **Features**: - Raycast from component transform (attach as child of motion controller) - Automatic detection of `AMosisPhoneActor` hits - Touch state machine: Down/Move/Up events - Optional Enhanced Input binding via `TriggerAction` property - Manual control via `SetTriggerPressed(bool)` - Debug ray visualization **Blueprint Setup**: 1. Add `MosisPointerComponent` as child of `MotionControllerComponent` 2. Set `TriggerAction` to your trigger input action (optional) 3. Touch events are sent automatically when pointing at phone and triggering **C++ Setup**: ```cpp // In VR Pawn header UPROPERTY(VisibleAnywhere) UMosisPointerComponent* RightPointer; // In constructor RightPointer = CreateDefaultSubobject(TEXT("RightPointer")); RightPointer->SetupAttachment(RightMotionController); // In BeginPlay (if using Enhanced Input) RightPointer->TriggerAction = TriggerInputAction; ``` **Manual Control (without Enhanced Input)**: ```cpp // In your input handling code void AMyVRPawn::OnTriggerPressed() { RightPointer->SetTriggerPressed(true); } void AMyVRPawn::OnTriggerReleased() { RightPointer->SetTriggerPressed(false); } ``` **Component Properties**: | Property | Type | Default | Description | |----------|------|---------|-------------| | `RayLength` | float | 500.0 | Maximum ray length in cm | | `bShowDebugRay` | bool | false | Draw debug visualization | | `DebugRayColor` | FLinearColor | Red | Ray color when not hitting | | `DebugRayHitColor` | FLinearColor | Green | Ray color when hitting phone | | `TraceChannel` | ECollisionChannel | Visibility | Collision channel for raycast | | `TriggerAction` | UInputAction* | nullptr | Enhanced Input action for trigger | **Blueprint Functions**: | Function | Returns | Description | |----------|---------|-------------| | `IsPointingAtPhone()` | bool | True if ray hits a phone actor | | `GetTargetPhone()` | AMosisPhoneActor* | Currently targeted phone (or nullptr) | | `GetHitLocation()` | FVector | World location of ray hit | | `GetRayOrigin()` | FVector | Ray start position | | `GetRayDirection()` | FVector | Ray direction vector | | `IsTriggerPressed()` | bool | Current trigger state | | `SetTriggerPressed(bool)` | void | Manual trigger control | **Touch State Machine**: ``` Idle ──[raycast hit]──► Hover ──[trigger]──► Pressed ▲ │ │ │ │ raycast miss │ trigger release │ ▼ ▼ └──────────────────── Idle ◄──────────────────── Events: - Idle → Pressed: SendTouch(Down) - Pressed + moving: SendTouch(Move) - Pressed → Idle: SendTouch(Up) ``` ### MosisVR (Unity 6000.3.2f1) **Location**: `D:\Dev\Mosis\MosisVR\Packages\com.omixlab.mosis_sdk\` #### Direct APK Build (Recommended) ```batch "C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^ -batchmode -quit -nographics ^ -projectPath "D:\Dev\Mosis\MosisVR" ^ -executeMethod BuildScript.BuildAndroidDirectCI ^ -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk" ``` #### Export + Gradle Build For more control, export a Gradle project then build separately: ```batch :: Step 1: Export from Unity "C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^ -batchmode -quit -nographics ^ -projectPath "D:\Dev\Mosis\MosisVR" ^ -executeMethod BuildScript.BuildAndroidCI ^ -export true ^ -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR" :: Step 2: Build with Gradle cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR gradle assembleRelease :: APK at: launcher\build\outputs\apk\release\launcher-release.apk ``` #### Unity Editor Manual Build 1. File > Build Settings > Android 2. Player Settings: IL2CPP, ARM64, Vulkan + OpenGLES3 3. For direct APK: Uncheck "Export Project", click Build 4. For export: Check "Export Project", click Export #### Native Plugin Build (Manual) The native plugin builds automatically via CMake during Unity's build. To rebuild manually: ```batch cd Packages/com.omixlab.mosis_sdk/Plugins/Android/cpp cmake -B build -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29 cmake --build build ``` ### Device Testing (Both Engines) ```bash # Install MosisService first adb install -r MosisService-debug.apk # Install game client adb install -r MosisUnreal-arm64.apk # or MosisVR.apk # Launch service adb shell am start -n com.omixlab.mosis/.MainActivity # Launch client adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity # or for Unity: adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity # Monitor all Mosis logs adb logcat -s MosisSDK MosisTest RMLUI Vulkan ``` ## Vulkan HardwareBuffer Import Both game engines use Vulkan to import AHardwareBuffer from MosisService. ### Required Vulkan Extensions ``` VK_ANDROID_external_memory_android_hardware_buffer VK_KHR_external_memory VK_KHR_dedicated_allocation ``` ### Import Pattern ```cpp // 1. Query buffer properties VkAndroidHardwareBufferPropertiesANDROID props = { .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID }; VkAndroidHardwareBufferFormatPropertiesANDROID formatProps = { .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID }; props.pNext = &formatProps; vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &props); // 2. Create image with external memory VkExternalMemoryImageCreateInfo extInfo = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO, .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID }; AHardwareBuffer_Desc desc; AHardwareBuffer_describe(buffer, &desc); VkImageCreateInfo imageInfo = { .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .pNext = &extInfo, .imageType = VK_IMAGE_TYPE_2D, .format = formatProps.format, .extent = {desc.width, desc.height, 1}, .mipLevels = 1, .arrayLayers = 1, .samples = VK_SAMPLE_COUNT_1_BIT, .tiling = VK_IMAGE_TILING_OPTIMAL, .usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT, .sharingMode = VK_SHARING_MODE_EXCLUSIVE }; vkCreateImage(device, &imageInfo, nullptr, &image); // 3. Import memory from HardwareBuffer VkImportAndroidHardwareBufferInfoANDROID importInfo = { .sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID, .buffer = buffer }; VkMemoryDedicatedAllocateInfo dedicatedInfo = { .sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO, .pNext = &importInfo, .image = image }; VkMemoryAllocateInfo allocInfo = { .sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .pNext = &dedicatedInfo, .allocationSize = props.allocationSize, .memoryTypeIndex = FindMemoryType(props.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) }; vkAllocateMemory(device, &allocInfo, nullptr, &memory); vkBindImageMemory(device, image, memory, 0); ``` ### Synchronization Currently using CPU synchronization (`vkQueueWaitIdle` / `glFinish`). Future improvement: use Vulkan semaphores for GPU-GPU sync. ### Double Buffering The imported image is copied to a local texture each frame to prevent data races with MosisService rendering. ## Developer Portal The Developer Portal is a web application for app developers to publish and manage their Mosis apps. Planning documents are in `DEV_PORTAL_M01-M12.md` files. ### Architecture Decisions | Component | Technology | Rationale | |-----------|------------|-----------| | Backend | Go 1.22+ | Simple, fast, single binary deployment | | Router | Chi | Lightweight, idiomatic Go HTTP routing | | Database | SQLite + Litestream | Zero-ops, continuous backup to NAS storage | | Frontend | htmx + Go templates | Server-rendered, minimal JS, fast | | Storage | Synology NAS filesystem | Self-hosted, local volume mounts | | Auth | OAuth2 (GitHub/Google) + JWT | golang-jwt for tokens, Ed25519 for signing | | CLI | Go + Cobra | Cross-platform single binary | | Docs | Hugo + Docsy | Static site served from /docs/ | ### Deployment Target Self-hosted on Synology NAS via Docker: ``` /volume1/mosis/ ├── data/ │ ├── portal.db # Main SQLite database │ └── telemetry.db # Separate telemetry database ├── packages/ # Uploaded app packages │ └── {dev_id}/{app_id}/{version}/ ├── backups/ # Litestream replicas └── docs/ # Hugo static site output ``` ### Milestone Documents | File | Topic | Status | |------|-------|--------| | DEV_PORTAL_M01_OVERVIEW.md | Project overview | Decided | | DEV_PORTAL_M02_WEB_STACK.md | Go + Chi + htmx | Decided | | DEV_PORTAL_M03_DATABASE.md | SQLite + Litestream | Decided | | DEV_PORTAL_M04_AUTH.md | OAuth2 + JWT + Ed25519 | Decided | | DEV_PORTAL_M05_FRONTEND.md | htmx + Go templates | Decided | | DEV_PORTAL_M06_API.md | REST API design | Decided | | DEV_PORTAL_M07_STORAGE.md | NAS filesystem | Decided | | DEV_PORTAL_M08_TELEMETRY.md | SQLite + background workers | Decided | | DEV_PORTAL_M09_REVIEW.md | Go validation workers | Decided | | DEV_PORTAL_M10_DEVICE.md | C++ AppManager | Decided | | DEV_PORTAL_M11_CLI.md | Go + Cobra CLI | Decided | | DEV_PORTAL_M12_DOCS.md | Hugo + Docsy | Decided | ### Key Design Principles 1. **Single container** - Portal runs as one Docker container on NAS 2. **No external services** - SQLite, local filesystem, Pagefind search 3. **Pure Go** - `modernc.org/sqlite` (no CGO required) 4. **Server-rendered** - htmx for interactivity, no SPA framework 5. **Background workers** - Go goroutines for aggregation/cleanup jobs ## App Management System (M10) The device-side app management system handles installation, updates, and launching of third-party apps. ### Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ AppManager │ │ - Install/Uninstall apps from .mosis packages │ │ - Track installed apps in JSON registry │ │ - Manage app data/cache directories │ └─────────────────────────┬───────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ LuaSandboxManager │ │ - StartApp/StopApp lifecycle │ │ - Per-app isolated Lua environments │ │ - Resource limits (memory, CPU, timers) │ └─────────────────────────────────────────────────────────────┘ ``` ### App Package Format (.mosis) Apps are distributed as ZIP files with `.mosis` extension: ``` myapp.mosis ├── manifest.json # Required: app metadata ├── main.rml # Entry point (RmlUi document) ├── styles.rcss # Stylesheets ├── scripts/ # Lua scripts │ └── app.lua └── assets/ # Icons, images, etc. ``` ### Manifest Format ```json { "id": "com.example.myapp", "name": "My App", "version": "1.0.0", "version_code": 1, "entry": "main.rml", "icon": "icon.tga", "description": "App description", "developer": { "name": "Developer Name", "email": "dev@example.com" }, "permissions": [ "network", "storage", "camera" ], "min_api_version": 1 } ``` ### App Lifecycle ```cpp // Install from local file app_manager->InstallFromFile("/path/to/app.mosis", [](auto progress) { LOG_INFO("Install progress: %s %.0f%%", InstallProgress::StageName(progress.stage), progress.progress * 100); }); // Launch app (starts sandbox) app_manager->LaunchApp("com.example.myapp"); // Check if running bool running = app_manager->IsAppRunning("com.example.myapp"); // Stop app (cleanup sandbox) app_manager->StopApp("com.example.myapp"); // Uninstall (stops if running, removes files) app_manager->Uninstall("com.example.myapp", false); // keep_data=false ``` ### Directory Structure ``` /data/data/com.omixlab.mosis/files/ ├── apps/ │ └── com.example.myapp/ │ ├── package/ # Extracted app files │ ├── data/ # App persistent data (VirtualFS) │ ├── cache/ # App cache (clearable) │ └── db/ # SQLite databases ├── downloads/ # Temporary download location ├── backups/ # App data backups └── config/ └── apps.json # Installed apps registry ``` ## Lua Sandbox System The sandbox provides secure, isolated Lua environments for third-party apps. ### Security Features | Feature | Implementation | |---------|----------------| | Dangerous globals removed | `os`, `io`, `loadfile`, `dofile`, `debug` | | Memory limits | Configurable per-app (default 10MB) | | CPU limits | Instruction counting with timeout | | Bytecode rejected | Only source code allowed | | Metatables protected | Cannot modify string/table metatables | | Path traversal blocked | `../` and absolute paths rejected | ### Permission Categories | Category | Auto-Grant | Examples | |----------|------------|----------| | Normal | Yes | `storage`, `network` | | Dangerous | User consent | `camera`, `microphone`, `location`, `contacts` | | Signature | System apps only | `system_settings`, `install_packages` | ### Available APIs **Core APIs** (always available): ```lua -- Timers local id = setTimeout(function() end, 1000) clearTimeout(id) local id = setInterval(function() end, 500) clearInterval(id) -- JSON local obj = json.decode('{"key": "value"}') local str = json.encode({key = "value"}) -- Crypto local bytes = crypto.randomBytes(16) local hash = crypto.sha256("data") local hmac = crypto.hmac("sha256", "key", "data") ``` **Storage APIs** (requires `storage` permission): ```lua -- Virtual filesystem (sandboxed to app directory) fs.write("data.txt", "content") local content = fs.read("data.txt") local files = fs.list("/") local stat = fs.stat("data.txt") fs.delete("data.txt") -- SQLite database local db = database.open("mydb") db:execute("CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, name TEXT)") db:execute("INSERT INTO items (name) VALUES (?)", {"item1"}) local rows = db:query("SELECT * FROM items WHERE id = ?", {1}) ``` **Network APIs** (requires `network` permission): ```lua -- HTTP (HTTPS only, private IPs blocked) local response = http.get("https://api.example.com/data") local response = http.post("https://api.example.com/data", { headers = {["Content-Type"] = "application/json"}, body = json.encode({key = "value"}) }) -- WebSocket local ws = websocket.connect("wss://example.com/ws") ws:send("message") ws:onMessage(function(data) end) ws:close() ``` **Hardware APIs** (requires dangerous permissions + user gesture): ```lua -- Camera (requires camera permission) camera.start(function(frame) end) camera.stop() -- Microphone (requires microphone permission) microphone.start(function(samples) end) microphone.stop() -- Location (requires location permission) location.getCurrentPosition(function(pos) print(pos.latitude, pos.longitude) end) -- Sensors (requires sensors permission) sensors.subscribe("accelerometer", function(data) print(data.x, data.y, data.z) end) ``` ### Running Sandbox Tests ```bash cd sandbox-test cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake cmake --build build --config Debug ./build/Debug/sandbox-test.exe # Output: 149 tests, all passing ``` ### Test Categories | Category | Tests | Description | |----------|-------|-------------| | Security | 11 | Globals removal, bytecode, metatables | | Resources | 8 | Memory, CPU limits, instruction counting | | Permissions | 7 | Normal/dangerous/signature grants | | Rate Limiting | 6 | API call throttling | | Timers | 7 | setTimeout/setInterval behavior | | JSON | 5 | Encode/decode, depth limits | | Crypto | 4 | Random, SHA256, HMAC | | VirtualFS | 8 | Read/write, quotas, traversal | | Database | 8 | SQLite operations, injection prevention | | Network | 8 | URL validation, private IP blocking | | WebSocket | 7 | Connection limits, message size | | Hardware | 42 | Camera, mic, location, sensors, bluetooth | | IPC | 7 | Message bus between apps | | Integration | 9 | Full app lifecycle | | Fuzzing | 3 | Random input crash testing |