# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Company **OmixLab LTD** - Package namespace: `com.omixlab` ## Git Commit Guidelines **IMPORTANT**: When creating git commits: - **DO NOT** add yourself as a co-author (no `Co-Authored-By` lines) - **Commit messages must be a single line** - keep it concise and descriptive ## Project Overview Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users can interact with inside VR environments, with real smartphone functionality. ### Current Status | Component | Status | Notes | |-----------|--------|-------| | MosisService | ✅ Working | RmlUi rendering, touch input, navigation | | Desktop Designer | ✅ Working | Hot-reload, hierarchy dump, recording | | Designer Tests | ✅ 5/5 Passing | Navigation tests automated | | MosisVR (Unity) | ✅ Building | OpenGL backend working, Vulkan in progress | | MosisUnreal | ✅ Working | Vulkan texture import via UE5 RHI, phone actor with mesh | ### Project Components | Component | Location | Purpose | |-----------|----------|---------| | Android Service | `src/main/` | Native service running RmlUi renderer | | Desktop Designer | `designer/` | UI development with hot-reload | | Designer Tests | `designer-test/` | Automated UI testing framework | | Sandbox Tests | `sandbox-test/` | Lua sandbox security tests | | Lua Sandbox | `src/main/cpp/sandbox/` | Per-app Lua isolation | | UI Assets | `src/main/assets/` | Shared RML/RCSS/Lua assets | ## Build Commands ### Android (Gradle) ```bash # Build entire project ./gradlew build # Build debug APK ./gradlew assembleDebug # Build release APK ./gradlew assembleRelease # Clean build outputs ./gradlew clean # Build with verbose output ./gradlew build --info --stacktrace # Run lint checks ./gradlew lint # Run unit tests ./gradlew test # Run connected device tests ./gradlew connectedAndroidTest ``` ### Desktop Designer (CMake) ```bash # Configure (from designer/ folder) cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake # Build cmake --build build --config Debug # Run with hot-reload ./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml # Run with logging and hierarchy dump ./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml --log output.log --hierarchy hierarchy.json ``` ### Designer Tests (CMake) ```bash # Configure (from designer-test/ folder) cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake # Build cmake --build build --config Debug # Run tests ./build/Debug/designer-test.exe ``` ### Sandbox Security Tests (CMake) ```bash # Configure (from sandbox-test/ folder) cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake # Build cmake --build build --config Debug # Run all tests (uber command) ./run_tests.bat # Or run directly ./build/Debug/sandbox-test.exe # Run specific test ./build/Debug/sandbox-test.exe --test DangerousGlobals ./build/Debug/sandbox-test.exe --test Memory ./build/Debug/sandbox-test.exe --test CPU ``` ## Environment Requirements Required environment variables: - `ANDROID_HOME` - Android SDK path - `ANDROID_NDK_HOME` - Android NDK path (version 29.0.14206865) - `VCPKG_ROOT` - vcpkg package manager root ## Architecture Overview MosisService is an Android application combining Kotlin UI with native C++ libraries for UI rendering via Android's Binder IPC system. ### Two Native Libraries **mosis-service** (`libmosis-service.so`): - Main Android Binder service implementation - Implements `IMosisService.aidl` interface for touch events and initialization - Contains the Kernel rendering engine with RmlUi integration - Links against RmlUi for HTML/CSS-like UI rendering **mosis-test** (`libmosis-test.so`): - Test/rendering client implementation - Implements `IMosisListener.aidl` for receiving callbacks - OpenGL ES 2.0 rendering pipeline using GLAD ### IPC Flow ``` Kotlin NativeService → JNI → mosis-service (IMosisService) ↓ IMosisListener callbacks ↓ mosis-test (rendering client) ``` ### Key Interfaces (AIDL) `IMosisService`: `initOS()`, `onTouchDown()`, `onTouchMove()`, `onTouchUp()` `IMosisListener` (oneway/async): `onServiceInitialized()`, `onBufferAvailable()`, `onFrameAvailable()` ### Native Code Structure (src/main/cpp/) - `kernel.cpp` - Core rendering engine, RmlUi integration, event processing - `mosis-service.cpp` - Binder service implementation, JNI entry points - `mosis-test.cpp` - Test client implementation - `egl_context.cpp` - OpenGL ES context management - `render_target.cpp` - Framebuffer and buffer management - `RmlUi_Renderer_GL3.cpp` - RmlUi OpenGL renderer backend - `assets_manager.cpp` - Android AssetManager integration ## Code Style - C++23 standard with modern features (std::span, std::format) - PascalCase for classes/functions, camelCase for variables - RAII principles with smart pointers - Kotlin code follows Android conventions ## Dependencies - vcpkg manages native dependencies (RmlUi, GLFW, Freetype, Lua, libpng, nlohmann-json) - CMake build system with vcpkg toolchain integration - Android target architecture: arm64-v8a only - Desktop target: Windows x64 (MSVC) ## Desktop Designer The desktop designer (`designer/`) provides rapid UI development with: - **Hot-reload**: Automatically reloads when RML/RCSS/Lua files change - **UI Hierarchy Dumping**: Exports element tree to JSON for inspection - **Screenshot Capture**: PNG export via F12 key - **Logging**: Detailed output for debugging navigation and events - **Action Recording**: Record mouse/keyboard interactions to JSON - **Action Playback**: Replay recorded interactions with timing ### Key Files | File | Purpose | |------|---------| | `designer/src/main.cpp` | Main entry point, GLFW window, event loop | | `designer/src/desktop_kernel.cpp` | RmlUi context management, rendering | | `designer/src/testing/ui_inspector.cpp` | UI hierarchy JSON export | | `designer/src/testing/visual_capture.cpp` | PNG screenshot capture and comparison | | `designer/src/testing/action_recorder.cpp` | Record user interactions to JSON | | `designer/src/testing/action_player.cpp` | Playback recorded actions | | `designer/src/backend/RmlUi_Backend_GLFW_GL3.cpp` | GLFW backend with input hooks | ### Command Line Options ``` --log Write logs to file --hierarchy Dump UI hierarchy JSON each frame --dump Single-shot dump mode (screenshot + hierarchy) --record Enable recording mode (F5 to start/stop) --playback Play back recorded actions from JSON ``` ### Keyboard Controls | Key | Function | |-----|----------| | F5 | Start/stop recording (when --record is enabled) | | F6 | Pause/resume playback (when --playback is enabled) | | F12 | Take screenshot | ## Automated Testing Framework The designer-test (`designer-test/`) provides automated UI testing: ### Test Architecture 1. **WindowController**: Finds designer window, sends mouse/keyboard events via Windows SendInput API 2. **HierarchyReader**: Parses UI hierarchy JSON to find elements by ID/class (with retry logic and exponential backoff) 3. **LogParser**: Monitors log file for navigation events 4. **TestRunner**: Orchestrates test execution, reports results 5. **UIInspector**: Dumps UI hierarchy with atomic writes (temp file + rename pattern) ### Key Implementation Details - **Path Normalization**: RmlUi uses `|` instead of `:` in Windows paths (e.g., `D|\Dev\...`). The UIInspector normalizes paths for correct document matching. - **Atomic File Writes**: Hierarchy files are written to `.tmp` then renamed to prevent partial reads. - **Retry with Backoff**: HierarchyReader retries up to 10 times with exponential backoff (30ms base) and validates JSON completeness. - **Dynamic Back Button**: `GoHome()` finds back buttons from hierarchy by class (`app-bar-nav` or `browser-nav-btn`) instead of fixed coordinates. ### Writing Tests ```cpp // Find element by ID and click it bool ClickById(TestContext& ctx, const std::string& id) { ctx.hierarchy.Reload(); auto element = ctx.hierarchy.FindById(id); if (!element) return false; int x = element->bounds.centerX(); int y = element->bounds.centerY(); ScaleToPhysical(ctx, x, y); // Convert logical to physical coords ctx.window.SendClick(x, y); return true; } // Test example bool TestNavigateToDialer(TestContext& ctx) { GoHome(ctx); ctx.log.Clear(); if (!ClickById(ctx, "dock-phone")) return false; std::this_thread::sleep_for(std::chrono::milliseconds(1000)); ctx.log.Reload(); return ctx.log.Contains("Loaded screen: apps/dialer/dialer.rml"); } ``` ### Test Output Tests produce JSON results at `test_results.json`: ```json { "name": "Mosis Designer UI Tests", "summary": {"passed": 5, "failed": 0, "total": 5}, "tests": [ {"name": "Navigate to Dialer", "status": "passed", "duration_ms": 3500} ] } ``` ## UI Assets Structure All UI assets are in `src/main/assets/`: ``` assets/ ├── apps/ # System apps (RML documents) │ ├── home/home.rml # Home screen launcher │ ├── dialer/dialer.rml # Phone dialer │ ├── messages/ # Messages app │ ├── contacts/ # Contacts app │ ├── settings/ # Settings app │ └── browser/ # Web browser ├── ui/ # Shared stylesheets │ ├── html.rcss # Base HTML element styles │ ├── theme.rcss # Design tokens (colors, typography) │ └── components.rcss # Reusable UI components ├── scripts/ # Lua scripts │ └── navigation.lua # Navigation system ├── icons/ # TGA icon files (24x24, 32x32) └── fonts/ # TTF fonts (LatoLatin, Roboto) ``` ### Navigation System Navigation is handled by `scripts/navigation.lua`: ```lua -- Navigate to a screen navigateTo('dialer') -- Push to history, animate forward -- Go back goBack() -- Pop from history, animate back -- Go home goHome() -- Clear history, return to home ``` ### Element IDs for Testing Key elements have IDs for automated testing: | ID | Location | Purpose | |----|----------|---------| | `dock-phone` | home.rml | Phone dock icon | | `dock-messages` | home.rml | Messages dock icon | | `dock-contacts` | home.rml | Contacts dock icon | | `dock-browser` | home.rml | Browser dock icon | | `app-settings` | home.rml | Settings app icon | ### CSS Classes for Navigation Back buttons use `app-bar-nav` class for automated GoHome: ```html
``` Browser uses `browser-nav-btn` class for its toolbar back button: ```html
``` The test framework's `FindBackButton()` searches for both classes to handle all screen layouts. ## Material Design Resources Material Design icons and components are available in the MosisDesigner repository: ### Material Design Icons **Location**: `D:\Dev\Mosis\MosisDesigner\material-design-icons` A comprehensive icon library from Google with 2000+ icons across 20 categories: | Category | Examples | |----------|----------| | action | home, search, settings, delete, info | | alert | error, warning, notification | | av | play, pause, volume, mic | | communication | phone, message, email, contacts | | content | add, remove, copy, paste | | device | battery, wifi, bluetooth, gps | | editor | format, text, color, brush | | file | folder, attachment, download, upload | | hardware | keyboard, mouse, phone, tablet | | home | lightbulb, thermostat, security | | image | camera, photo, filter, tune | | maps | location, directions, navigation | | navigation | arrow, chevron, menu, close | | notification | sync, update, event | | places | hotel, restaurant, airport | | search | search variants | | social | share, person, group, notifications | | toggle | star, checkbox, radio | **Available Formats**: - `src/` - SVG source files organized by category - `png/` - PNG files at multiple DPIs (24dp, 36dp, 48dp) - `font/` - Icon fonts (WOFF, TTF) - `symbols/` - Material Symbols variable font (newer) - `variablefont/` - Variable font files **Icon Styles**: - Outlined (default) - Filled - Rounded - Sharp - Two-tone (Material Icons only) ### Material Design Lite **Location**: `D:\Dev\Mosis\MosisDesigner\material-design-lite` CSS/JS component library implementing Material Design (reference implementation): | Directory | Contents | |-----------|----------| | `src/` | SASS source for components | | `docs/` | Component documentation | | `templates/` | Page templates | **Key Components** (for design reference): - Buttons (raised, flat, FAB) - Cards - Dialogs - Lists - Menus - Navigation drawers - Progress indicators - Sliders - Snackbars - Tables - Tabs - Text fields - Tooltips ### Using Icons in Mosis 1. **Find icon** at https://fonts.google.com/icons 2. **Export SVG** from `material-design-icons/src///` 3. **Convert to TGA** using image tool (24x24 or 32x32, RGBA) 4. **Place in** `src/main/assets/icons/` 5. **Reference in RML**: `` ## Android Device Testing ### Prerequisites ```bash # Check connected device adb devices -l # Verify Mosis app is installed adb shell pm list packages | grep mosis ``` ### Build and Install ```bash # Build debug APK ./gradlew assembleDebug # Install on device adb install -r build/outputs/apk/debug/MosisService-debug.apk # Launch the app adb shell am start -n com.omixlab.mosis/.MainActivity ``` ### Run Gradle Connected Tests ```bash # Run all connected Android tests ./gradlew connectedAndroidTest ``` ### Event Injection via ADB Inject touch events for automated testing: ```bash # Click at normalized coordinates (0.0-1.0) adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.5 --ef y 0.5 # Touch down adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "down" --ef x 0.2 --ef y 0.9 # Touch up adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "up" --ef x 0.2 --ef y 0.9 ``` ### Dock Element Coordinates (Normalized) | Element | X | Y | |---------|---|---| | dock-phone | 0.16 | 0.97 | | dock-messages | 0.39 | 0.97 | | dock-contacts | 0.61 | 0.97 | | dock-browser | 0.84 | 0.97 | | back-button | 0.10 | 0.05 | ### Full Navigation Test Sequence ```bash # Clear logs and run navigation test sequence adb logcat -c # Click Phone dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.16 --ef y 0.97 sleep 2 # Click back to return home adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.1 --ef y 0.05 sleep 2 # Click Messages dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.39 --ef y 0.97 sleep 2 # Click back to return home adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.1 --ef y 0.05 sleep 2 # Click Contacts dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.61 --ef y 0.97 sleep 2 # Click back to return home adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.1 --ef y 0.05 sleep 2 # Click Browser dock icon adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \ --es touch_type "click" --ef x 0.84 --ef y 0.97 ``` ### Reading Logs ```bash # Filter for Mosis logs adb logcat -s MosisTest ServiceTester RMLUI # Filter for navigation events adb logcat -d | grep -iE "navigat|loaded|goBack|rml" # Save to file adb logcat -s MosisTest > mosis-log.txt # Clear logs adb logcat -c ``` ### Expected Log Output Successful navigation shows these log patterns: ``` RMLUI: navigateTo called with: dialer Loading screen: apps/dialer/dialer.rml RMLUI: Navigated to: dialer (history depth: 1) RMLUI: goBack called (history depth: 1) Loading screen: apps/home/home.rml RMLUI: Back to: home ``` ## Documentation Guidelines **IMPORTANT**: Always document progress and new commands to avoid rediscovery. ### Where to Put Documentation | Content Type | Location | |--------------|----------| | General concepts, architecture | `MosisService/CLAUDE.md` (this file) | | Unreal plugin docs | `MosisUnreal/Plugins/MosisSDK/README.md` | | Unity package docs | `MosisVR/Packages/com.omixlab.mosis_sdk/README.md` | | Project-specific build commands | Each project's own docs | **DO NOT** put documentation in the root `D:\Dev\Mosis\` directory - it is not versioned. ### What to Document 1. **Build commands** - Every new build command discovered or created 2. **Environment setup** - Required environment variables, SDK versions 3. **Architecture decisions** - Why something was done a certain way 4. **Issues and solutions** - Problems encountered and how they were fixed 5. **File locations** - Where important files are and what they do ### When to Document - After completing a milestone or feature - When discovering new build commands - When fixing a non-obvious issue - When adding new dependencies or requirements ## Game Engine Integrations MosisService provides a virtual phone that game engines can display and interact with. ### Integration Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ MosisService │ │ (OpenGL ES rendering → AHardwareBuffer) │ └───────────────────────────┬─────────────────────────────────────┘ │ Binder IPC + Shared Memory ┌─────────────────┴─────────────────┐ ▼ ▼ ┌─────────────────────┐ ┌─────────────────────┐ │ MosisUnreal │ │ MosisVR │ │ (UE5.5 Plugin) │ │ (Unity Package) │ │ Vulkan Import │ │ Vulkan/OpenGL │ └─────────────────────┘ └─────────────────────┘ ``` ### MosisUnreal (Unreal Engine 5.5) **Location**: `D:\Dev\Mosis\MosisUnreal\Plugins\MosisSDK\` #### Build Commands ```batch :: Windows Editor Build "D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^ MosisUnrealEditor Win64 Development ^ -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" :: Android APK Build "D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^ BuildCookRun ^ -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^ -platform=Android -clientconfig=Development ^ -build -cook -stage -pak -package -noP4 :: Clean Build (delete these folders first) rmdir /s /q "Intermediate\Build" rmdir /s /q "Binaries" ``` #### Output Files | Build | Output | |-------|--------| | Windows Editor | `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll` | | Android APK | `Binaries/Android/MosisUnreal-arm64.apk` | | Android OBB | `Binaries/Android/main.1.com.omixlab.MosisUnreal.obb` | #### Requirements - Android SDK Platform 36 (for AIDL binder headers) - Android Build Tools 36.1.0 (for AIDL compiler) - `ANDROID_HOME` environment variable set #### Quest Deployment **IMPORTANT**: Git Bash on Windows converts Unix paths to Windows paths. Use `MSYS_NO_PATHCONV=1` prefix for all adb push commands. ```bash # Set Quest device ID (check with: adb devices -l) QUEST=2G0YC5ZF7W01T0 # Install APK adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk # Create OBB temp directory adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal" # Push OBB to temp location (MUST use MSYS_NO_PATHCONV=1) MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/ # Move OBB to final location adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal" adb -s $QUEST shell "mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/" # Start MosisService first, then MosisUnreal adb -s $QUEST shell am start -n com.omixlab.mosis/.MainActivity sleep 3 adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity # Monitor logs adb -s $QUEST logcat -s MosisSDK MosisOS MosisTest ``` **Full rebuild and deploy sequence**: ```bash # 1. Build Android APK+OBB "D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun \ -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" \ -platform=Android -clientconfig=Development \ -build -cook -stage -pak -package -noP4 # 2. Stop app, push APK and OBB, restart QUEST=2G0YC5ZF7W01T0 adb -s $QUEST shell am force-stop com.omixlab.MosisUnreal adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal" MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/ adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal && mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/" adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity ``` **Common issues**: - App stuck on loading screen: OBB not pushed or wrong version - `adb push` path errors: Missing `MSYS_NO_PATHCONV=1` prefix - Service not connecting: Start `com.omixlab.mosis` before the game #### VR Pointer Interaction The MosisSDK includes `UMosisPointerComponent` for VR ray-based touch interaction. **Key Files**: - `Public/MosisPointerComponent.h` - Component header - `Private/MosisPointerComponent.cpp` - Implementation **Features**: - Raycast from component transform (attach as child of motion controller) - Automatic detection of `AMosisPhoneActor` hits - Touch state machine: Down/Move/Up events - Optional Enhanced Input binding via `TriggerAction` property - Manual control via `SetTriggerPressed(bool)` - Debug ray visualization **Blueprint Setup**: 1. Add `MosisPointerComponent` as child of `MotionControllerComponent` 2. Set `TriggerAction` to your trigger input action (optional) 3. Touch events are sent automatically when pointing at phone and triggering **C++ Setup**: ```cpp // In VR Pawn header UPROPERTY(VisibleAnywhere) UMosisPointerComponent* RightPointer; // In constructor RightPointer = CreateDefaultSubobject(TEXT("RightPointer")); RightPointer->SetupAttachment(RightMotionController); // In BeginPlay (if using Enhanced Input) RightPointer->TriggerAction = TriggerInputAction; ``` **Manual Control (without Enhanced Input)**: ```cpp // In your input handling code void AMyVRPawn::OnTriggerPressed() { RightPointer->SetTriggerPressed(true); } void AMyVRPawn::OnTriggerReleased() { RightPointer->SetTriggerPressed(false); } ``` **Component Properties**: | Property | Type | Default | Description | |----------|------|---------|-------------| | `RayLength` | float | 500.0 | Maximum ray length in cm | | `bShowDebugRay` | bool | false | Draw debug visualization | | `DebugRayColor` | FLinearColor | Red | Ray color when not hitting | | `DebugRayHitColor` | FLinearColor | Green | Ray color when hitting phone | | `TraceChannel` | ECollisionChannel | Visibility | Collision channel for raycast | | `TriggerAction` | UInputAction* | nullptr | Enhanced Input action for trigger | **Blueprint Functions**: | Function | Returns | Description | |----------|---------|-------------| | `IsPointingAtPhone()` | bool | True if ray hits a phone actor | | `GetTargetPhone()` | AMosisPhoneActor* | Currently targeted phone (or nullptr) | | `GetHitLocation()` | FVector | World location of ray hit | | `GetRayOrigin()` | FVector | Ray start position | | `GetRayDirection()` | FVector | Ray direction vector | | `IsTriggerPressed()` | bool | Current trigger state | | `SetTriggerPressed(bool)` | void | Manual trigger control | **Touch State Machine**: ``` Idle ──[raycast hit]──► Hover ──[trigger]──► Pressed ▲ │ │ │ │ raycast miss │ trigger release │ ▼ ▼ └──────────────────── Idle ◄──────────────────── Events: - Idle → Pressed: SendTouch(Down) - Pressed + moving: SendTouch(Move) - Pressed → Idle: SendTouch(Up) ``` ### MosisVR (Unity 6000.3.2f1) **Location**: `D:\Dev\Mosis\MosisVR\Packages\com.omixlab.mosis_sdk\` #### Direct APK Build (Recommended) ```batch "C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^ -batchmode -quit -nographics ^ -projectPath "D:\Dev\Mosis\MosisVR" ^ -executeMethod BuildScript.BuildAndroidDirectCI ^ -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk" ``` #### Export + Gradle Build For more control, export a Gradle project then build separately: ```batch :: Step 1: Export from Unity "C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^ -batchmode -quit -nographics ^ -projectPath "D:\Dev\Mosis\MosisVR" ^ -executeMethod BuildScript.BuildAndroidCI ^ -export true ^ -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR" :: Step 2: Build with Gradle cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR gradle assembleRelease :: APK at: launcher\build\outputs\apk\release\launcher-release.apk ``` #### Unity Editor Manual Build 1. File > Build Settings > Android 2. Player Settings: IL2CPP, ARM64, Vulkan + OpenGLES3 3. For direct APK: Uncheck "Export Project", click Build 4. For export: Check "Export Project", click Export #### Native Plugin Build (Manual) The native plugin builds automatically via CMake during Unity's build. To rebuild manually: ```batch cd Packages/com.omixlab.mosis_sdk/Plugins/Android/cpp cmake -B build -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29 cmake --build build ``` ### Device Testing (Both Engines) ```bash # Install MosisService first adb install -r MosisService-debug.apk # Install game client adb install -r MosisUnreal-arm64.apk # or MosisVR.apk # Launch service adb shell am start -n com.omixlab.mosis/.MainActivity # Launch client adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity # or for Unity: adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity # Monitor all Mosis logs adb logcat -s MosisSDK MosisTest RMLUI Vulkan ``` ## Vulkan HardwareBuffer Import Both game engines use Vulkan to import AHardwareBuffer from MosisService. ### Required Vulkan Extensions ``` VK_ANDROID_external_memory_android_hardware_buffer VK_KHR_external_memory VK_KHR_dedicated_allocation ``` ### Import Pattern ```cpp // 1. Query buffer properties VkAndroidHardwareBufferPropertiesANDROID props = { .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID }; VkAndroidHardwareBufferFormatPropertiesANDROID formatProps = { .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID }; props.pNext = &formatProps; vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &props); // 2. Create image with external memory VkExternalMemoryImageCreateInfo extInfo = { .sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO, .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID }; AHardwareBuffer_Desc desc; AHardwareBuffer_describe(buffer, &desc); VkImageCreateInfo imageInfo = { .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .pNext = &extInfo, .imageType = VK_IMAGE_TYPE_2D, .format = formatProps.format, .extent = {desc.width, desc.height, 1}, .mipLevels = 1, .arrayLayers = 1, .samples = VK_SAMPLE_COUNT_1_BIT, .tiling = VK_IMAGE_TILING_OPTIMAL, .usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT, .sharingMode = VK_SHARING_MODE_EXCLUSIVE }; vkCreateImage(device, &imageInfo, nullptr, &image); // 3. Import memory from HardwareBuffer VkImportAndroidHardwareBufferInfoANDROID importInfo = { .sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID, .buffer = buffer }; VkMemoryDedicatedAllocateInfo dedicatedInfo = { .sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO, .pNext = &importInfo, .image = image }; VkMemoryAllocateInfo allocInfo = { .sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .pNext = &dedicatedInfo, .allocationSize = props.allocationSize, .memoryTypeIndex = FindMemoryType(props.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) }; vkAllocateMemory(device, &allocInfo, nullptr, &memory); vkBindImageMemory(device, image, memory, 0); ``` ### Synchronization Currently using CPU synchronization (`vkQueueWaitIdle` / `glFinish`). Future improvement: use Vulkan semaphores for GPU-GPU sync. ### Double Buffering The imported image is copied to a local texture each frame to prevent data races with MosisService rendering.