999 lines
32 KiB
Markdown
999 lines
32 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Company
|
|
|
|
**OmixLab LTD** - Package namespace: `com.omixlab`
|
|
|
|
## Git Commit Guidelines
|
|
|
|
**IMPORTANT**: When creating git commits:
|
|
- **DO NOT** add yourself as a co-author (no `Co-Authored-By` lines)
|
|
- **Commit messages must be a single line** - keep it concise and descriptive
|
|
|
|
## Project Overview
|
|
|
|
Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users can interact with inside VR environments, with real smartphone functionality.
|
|
|
|
### Current Status
|
|
|
|
| Component | Status | Notes |
|
|
|-----------|--------|-------|
|
|
| MosisService | ✅ Working | RmlUi rendering, touch input, navigation |
|
|
| Desktop Designer | ✅ Working | Hot-reload, hierarchy dump, recording |
|
|
| Designer Tests | ✅ 5/5 Passing | Navigation tests automated |
|
|
| MosisVR (Unity) | ✅ Building | OpenGL backend working, Vulkan in progress |
|
|
| MosisUnreal | ✅ Working | Vulkan texture import via UE5 RHI, phone actor with mesh |
|
|
|
|
### Project Components
|
|
|
|
| Component | Location | Purpose |
|
|
|-----------|----------|---------|
|
|
| Android Service | `src/main/` | Native service running RmlUi renderer |
|
|
| Desktop Designer | `designer/` | UI development with hot-reload |
|
|
| Designer Tests | `designer-test/` | Automated UI testing framework |
|
|
| Sandbox Tests | `sandbox-test/` | Lua sandbox security tests |
|
|
| Lua Sandbox | `src/main/cpp/sandbox/` | Per-app Lua isolation |
|
|
| UI Assets | `src/main/assets/` | Shared RML/RCSS/Lua assets |
|
|
|
|
## Build Commands
|
|
|
|
### Android (Gradle)
|
|
|
|
```bash
|
|
# Build entire project
|
|
./gradlew build
|
|
|
|
# Build debug APK
|
|
./gradlew assembleDebug
|
|
|
|
# Build release APK
|
|
./gradlew assembleRelease
|
|
|
|
# Clean build outputs
|
|
./gradlew clean
|
|
|
|
# Build with verbose output
|
|
./gradlew build --info --stacktrace
|
|
|
|
# Run lint checks
|
|
./gradlew lint
|
|
|
|
# Run unit tests
|
|
./gradlew test
|
|
|
|
# Run connected device tests
|
|
./gradlew connectedAndroidTest
|
|
```
|
|
|
|
### Desktop Designer (CMake)
|
|
|
|
```bash
|
|
# Configure (from designer/ folder)
|
|
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
|
|
|
|
# Build
|
|
cmake --build build --config Debug
|
|
|
|
# Run with hot-reload
|
|
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml
|
|
|
|
# Run with logging and hierarchy dump
|
|
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml --log output.log --hierarchy hierarchy.json
|
|
```
|
|
|
|
### Designer Tests (CMake)
|
|
|
|
```bash
|
|
# Configure (from designer-test/ folder)
|
|
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
|
|
|
|
# Build
|
|
cmake --build build --config Debug
|
|
|
|
# Run tests
|
|
./build/Debug/designer-test.exe
|
|
```
|
|
|
|
### Sandbox Security Tests (CMake)
|
|
|
|
```bash
|
|
# Configure (from sandbox-test/ folder)
|
|
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
|
|
|
|
# Build
|
|
cmake --build build --config Debug
|
|
|
|
# Run all tests (uber command)
|
|
./run_tests.bat
|
|
|
|
# Or run directly
|
|
./build/Debug/sandbox-test.exe
|
|
|
|
# Run specific test
|
|
./build/Debug/sandbox-test.exe --test DangerousGlobals
|
|
./build/Debug/sandbox-test.exe --test Memory
|
|
./build/Debug/sandbox-test.exe --test CPU
|
|
```
|
|
|
|
## Environment Requirements
|
|
|
|
Required environment variables:
|
|
- `ANDROID_HOME` - Android SDK path
|
|
- `ANDROID_NDK_HOME` - Android NDK path (version 29.0.14206865)
|
|
- `VCPKG_ROOT` - vcpkg package manager root
|
|
|
|
## Architecture Overview
|
|
|
|
MosisService is an Android application combining Kotlin UI with native C++ libraries for UI rendering via Android's Binder IPC system.
|
|
|
|
### Two Native Libraries
|
|
|
|
**mosis-service** (`libmosis-service.so`):
|
|
- Main Android Binder service implementation
|
|
- Implements `IMosisService.aidl` interface for touch events and initialization
|
|
- Contains the Kernel rendering engine with RmlUi integration
|
|
- Links against RmlUi for HTML/CSS-like UI rendering
|
|
|
|
**mosis-test** (`libmosis-test.so`):
|
|
- Test/rendering client implementation
|
|
- Implements `IMosisListener.aidl` for receiving callbacks
|
|
- OpenGL ES 2.0 rendering pipeline using GLAD
|
|
|
|
### IPC Flow
|
|
|
|
```
|
|
Kotlin NativeService → JNI → mosis-service (IMosisService)
|
|
↓
|
|
IMosisListener callbacks
|
|
↓
|
|
mosis-test (rendering client)
|
|
```
|
|
|
|
### Key Interfaces (AIDL)
|
|
|
|
`IMosisService`: `initOS()`, `onTouchDown()`, `onTouchMove()`, `onTouchUp()`
|
|
|
|
`IMosisListener` (oneway/async): `onServiceInitialized()`, `onBufferAvailable()`, `onFrameAvailable()`
|
|
|
|
### Native Code Structure (src/main/cpp/)
|
|
|
|
- `kernel.cpp` - Core rendering engine, RmlUi integration, event processing
|
|
- `mosis-service.cpp` - Binder service implementation, JNI entry points
|
|
- `mosis-test.cpp` - Test client implementation
|
|
- `egl_context.cpp` - OpenGL ES context management
|
|
- `render_target.cpp` - Framebuffer and buffer management
|
|
- `RmlUi_Renderer_GL3.cpp` - RmlUi OpenGL renderer backend
|
|
- `assets_manager.cpp` - Android AssetManager integration
|
|
|
|
## Code Style
|
|
|
|
- C++23 standard with modern features (std::span, std::format)
|
|
- PascalCase for classes/functions, camelCase for variables
|
|
- RAII principles with smart pointers
|
|
- Kotlin code follows Android conventions
|
|
|
|
## Dependencies
|
|
|
|
- vcpkg manages native dependencies (RmlUi, GLFW, Freetype, Lua, libpng, nlohmann-json)
|
|
- CMake build system with vcpkg toolchain integration
|
|
- Android target architecture: arm64-v8a only
|
|
- Desktop target: Windows x64 (MSVC)
|
|
|
|
## Desktop Designer
|
|
|
|
The desktop designer (`designer/`) provides rapid UI development with:
|
|
|
|
- **Hot-reload**: Automatically reloads when RML/RCSS/Lua files change
|
|
- **UI Hierarchy Dumping**: Exports element tree to JSON for inspection
|
|
- **Screenshot Capture**: PNG export via F12 key
|
|
- **Logging**: Detailed output for debugging navigation and events
|
|
- **Action Recording**: Record mouse/keyboard interactions to JSON
|
|
- **Action Playback**: Replay recorded interactions with timing
|
|
|
|
### Key Files
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `designer/src/main.cpp` | Main entry point, GLFW window, event loop |
|
|
| `designer/src/desktop_kernel.cpp` | RmlUi context management, rendering |
|
|
| `designer/src/testing/ui_inspector.cpp` | UI hierarchy JSON export |
|
|
| `designer/src/testing/visual_capture.cpp` | PNG screenshot capture and comparison |
|
|
| `designer/src/testing/action_recorder.cpp` | Record user interactions to JSON |
|
|
| `designer/src/testing/action_player.cpp` | Playback recorded actions |
|
|
| `designer/src/backend/RmlUi_Backend_GLFW_GL3.cpp` | GLFW backend with input hooks |
|
|
|
|
### Command Line Options
|
|
|
|
```
|
|
--log <path> Write logs to file
|
|
--hierarchy <path> Dump UI hierarchy JSON each frame
|
|
--dump Single-shot dump mode (screenshot + hierarchy)
|
|
--record <path> Enable recording mode (F5 to start/stop)
|
|
--playback <path> Play back recorded actions from JSON
|
|
```
|
|
|
|
### Keyboard Controls
|
|
|
|
| Key | Function |
|
|
|-----|----------|
|
|
| F5 | Start/stop recording (when --record is enabled) |
|
|
| F6 | Pause/resume playback (when --playback is enabled) |
|
|
| F12 | Take screenshot |
|
|
|
|
## Automated Testing Framework
|
|
|
|
The designer-test (`designer-test/`) provides automated UI testing:
|
|
|
|
### Test Architecture
|
|
|
|
1. **WindowController**: Finds designer window, sends mouse/keyboard events via Windows SendInput API
|
|
2. **HierarchyReader**: Parses UI hierarchy JSON to find elements by ID/class (with retry logic and exponential backoff)
|
|
3. **LogParser**: Monitors log file for navigation events
|
|
4. **TestRunner**: Orchestrates test execution, reports results
|
|
5. **UIInspector**: Dumps UI hierarchy with atomic writes (temp file + rename pattern)
|
|
|
|
### Key Implementation Details
|
|
|
|
- **Path Normalization**: RmlUi uses `|` instead of `:` in Windows paths (e.g., `D|\Dev\...`). The UIInspector normalizes paths for correct document matching.
|
|
- **Atomic File Writes**: Hierarchy files are written to `.tmp` then renamed to prevent partial reads.
|
|
- **Retry with Backoff**: HierarchyReader retries up to 10 times with exponential backoff (30ms base) and validates JSON completeness.
|
|
- **Dynamic Back Button**: `GoHome()` finds back buttons from hierarchy by class (`app-bar-nav` or `browser-nav-btn`) instead of fixed coordinates.
|
|
|
|
### Writing Tests
|
|
|
|
```cpp
|
|
// Find element by ID and click it
|
|
bool ClickById(TestContext& ctx, const std::string& id) {
|
|
ctx.hierarchy.Reload();
|
|
auto element = ctx.hierarchy.FindById(id);
|
|
if (!element) return false;
|
|
|
|
int x = element->bounds.centerX();
|
|
int y = element->bounds.centerY();
|
|
ScaleToPhysical(ctx, x, y); // Convert logical to physical coords
|
|
ctx.window.SendClick(x, y);
|
|
return true;
|
|
}
|
|
|
|
// Test example
|
|
bool TestNavigateToDialer(TestContext& ctx) {
|
|
GoHome(ctx);
|
|
ctx.log.Clear();
|
|
|
|
if (!ClickById(ctx, "dock-phone")) return false;
|
|
|
|
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
|
|
ctx.log.Reload();
|
|
return ctx.log.Contains("Loaded screen: apps/dialer/dialer.rml");
|
|
}
|
|
```
|
|
|
|
### Test Output
|
|
|
|
Tests produce JSON results at `test_results.json`:
|
|
```json
|
|
{
|
|
"name": "Mosis Designer UI Tests",
|
|
"summary": {"passed": 5, "failed": 0, "total": 5},
|
|
"tests": [
|
|
{"name": "Navigate to Dialer", "status": "passed", "duration_ms": 3500}
|
|
]
|
|
}
|
|
```
|
|
|
|
## UI Assets Structure
|
|
|
|
All UI assets are in `src/main/assets/`:
|
|
|
|
```
|
|
assets/
|
|
├── apps/ # System apps (RML documents)
|
|
│ ├── home/home.rml # Home screen launcher
|
|
│ ├── dialer/dialer.rml # Phone dialer
|
|
│ ├── messages/ # Messages app
|
|
│ ├── contacts/ # Contacts app
|
|
│ ├── settings/ # Settings app
|
|
│ └── browser/ # Web browser
|
|
├── ui/ # Shared stylesheets
|
|
│ ├── html.rcss # Base HTML element styles
|
|
│ ├── theme.rcss # Design tokens (colors, typography)
|
|
│ └── components.rcss # Reusable UI components
|
|
├── scripts/ # Lua scripts
|
|
│ └── navigation.lua # Navigation system
|
|
├── icons/ # TGA icon files (24x24, 32x32)
|
|
└── fonts/ # TTF fonts (LatoLatin, Roboto)
|
|
```
|
|
|
|
### Navigation System
|
|
|
|
Navigation is handled by `scripts/navigation.lua`:
|
|
|
|
```lua
|
|
-- Navigate to a screen
|
|
navigateTo('dialer') -- Push to history, animate forward
|
|
|
|
-- Go back
|
|
goBack() -- Pop from history, animate back
|
|
|
|
-- Go home
|
|
goHome() -- Clear history, return to home
|
|
```
|
|
|
|
### Element IDs for Testing
|
|
|
|
Key elements have IDs for automated testing:
|
|
|
|
| ID | Location | Purpose |
|
|
|----|----------|---------|
|
|
| `dock-phone` | home.rml | Phone dock icon |
|
|
| `dock-messages` | home.rml | Messages dock icon |
|
|
| `dock-contacts` | home.rml | Contacts dock icon |
|
|
| `dock-browser` | home.rml | Browser dock icon |
|
|
| `app-settings` | home.rml | Settings app icon |
|
|
|
|
### CSS Classes for Navigation
|
|
|
|
Back buttons use `app-bar-nav` class for automated GoHome:
|
|
```html
|
|
<div class="app-bar-nav btn-icon" onclick="goBack()">
|
|
<img src="../../icons/back.tga"/>
|
|
</div>
|
|
```
|
|
|
|
Browser uses `browser-nav-btn` class for its toolbar back button:
|
|
```html
|
|
<div class="app-bar-nav browser-nav-btn" onclick="goBack()">
|
|
<img src="../../icons/back.tga"/>
|
|
</div>
|
|
```
|
|
|
|
The test framework's `FindBackButton()` searches for both classes to handle all screen layouts.
|
|
|
|
## Material Design Resources
|
|
|
|
Material Design icons and components are available in the MosisDesigner repository:
|
|
|
|
### Material Design Icons
|
|
|
|
**Location**: `D:\Dev\Mosis\MosisDesigner\material-design-icons`
|
|
|
|
A comprehensive icon library from Google with 2000+ icons across 20 categories:
|
|
|
|
| Category | Examples |
|
|
|----------|----------|
|
|
| action | home, search, settings, delete, info |
|
|
| alert | error, warning, notification |
|
|
| av | play, pause, volume, mic |
|
|
| communication | phone, message, email, contacts |
|
|
| content | add, remove, copy, paste |
|
|
| device | battery, wifi, bluetooth, gps |
|
|
| editor | format, text, color, brush |
|
|
| file | folder, attachment, download, upload |
|
|
| hardware | keyboard, mouse, phone, tablet |
|
|
| home | lightbulb, thermostat, security |
|
|
| image | camera, photo, filter, tune |
|
|
| maps | location, directions, navigation |
|
|
| navigation | arrow, chevron, menu, close |
|
|
| notification | sync, update, event |
|
|
| places | hotel, restaurant, airport |
|
|
| search | search variants |
|
|
| social | share, person, group, notifications |
|
|
| toggle | star, checkbox, radio |
|
|
|
|
**Available Formats**:
|
|
- `src/` - SVG source files organized by category
|
|
- `png/` - PNG files at multiple DPIs (24dp, 36dp, 48dp)
|
|
- `font/` - Icon fonts (WOFF, TTF)
|
|
- `symbols/` - Material Symbols variable font (newer)
|
|
- `variablefont/` - Variable font files
|
|
|
|
**Icon Styles**:
|
|
- Outlined (default)
|
|
- Filled
|
|
- Rounded
|
|
- Sharp
|
|
- Two-tone (Material Icons only)
|
|
|
|
### Material Design Lite
|
|
|
|
**Location**: `D:\Dev\Mosis\MosisDesigner\material-design-lite`
|
|
|
|
CSS/JS component library implementing Material Design (reference implementation):
|
|
|
|
| Directory | Contents |
|
|
|-----------|----------|
|
|
| `src/` | SASS source for components |
|
|
| `docs/` | Component documentation |
|
|
| `templates/` | Page templates |
|
|
|
|
**Key Components** (for design reference):
|
|
- Buttons (raised, flat, FAB)
|
|
- Cards
|
|
- Dialogs
|
|
- Lists
|
|
- Menus
|
|
- Navigation drawers
|
|
- Progress indicators
|
|
- Sliders
|
|
- Snackbars
|
|
- Tables
|
|
- Tabs
|
|
- Text fields
|
|
- Tooltips
|
|
|
|
### Using Icons in Mosis
|
|
|
|
1. **Find icon** at https://fonts.google.com/icons
|
|
2. **Export SVG** from `material-design-icons/src/<category>/<name>/`
|
|
3. **Convert to TGA** using image tool (24x24 or 32x32, RGBA)
|
|
4. **Place in** `src/main/assets/icons/`
|
|
5. **Reference in RML**: `<img src="../../icons/<name>.tga"/>`
|
|
|
|
## Android Device Testing
|
|
|
|
### Prerequisites
|
|
|
|
```bash
|
|
# Check connected device
|
|
adb devices -l
|
|
|
|
# Verify Mosis app is installed
|
|
adb shell pm list packages | grep mosis
|
|
```
|
|
|
|
### Build and Install
|
|
|
|
```bash
|
|
# Build debug APK
|
|
./gradlew assembleDebug
|
|
|
|
# Install on device
|
|
adb install -r build/outputs/apk/debug/MosisService-debug.apk
|
|
|
|
# Launch the app
|
|
adb shell am start -n com.omixlab.mosis/.MainActivity
|
|
```
|
|
|
|
### Run Gradle Connected Tests
|
|
|
|
```bash
|
|
# Run all connected Android tests
|
|
./gradlew connectedAndroidTest
|
|
```
|
|
|
|
### Event Injection via ADB
|
|
|
|
Inject touch events for automated testing:
|
|
|
|
```bash
|
|
# Click at normalized coordinates (0.0-1.0)
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.5 --ef y 0.5
|
|
|
|
# Touch down
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "down" --ef x 0.2 --ef y 0.9
|
|
|
|
# Touch up
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "up" --ef x 0.2 --ef y 0.9
|
|
```
|
|
|
|
### Dock Element Coordinates (Normalized)
|
|
|
|
| Element | X | Y |
|
|
|---------|---|---|
|
|
| dock-phone | 0.16 | 0.97 |
|
|
| dock-messages | 0.39 | 0.97 |
|
|
| dock-contacts | 0.61 | 0.97 |
|
|
| dock-browser | 0.84 | 0.97 |
|
|
| back-button | 0.10 | 0.05 |
|
|
|
|
### Full Navigation Test Sequence
|
|
|
|
```bash
|
|
# Clear logs and run navigation test sequence
|
|
adb logcat -c
|
|
|
|
# Click Phone dock icon
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.16 --ef y 0.97
|
|
sleep 2
|
|
|
|
# Click back to return home
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.1 --ef y 0.05
|
|
sleep 2
|
|
|
|
# Click Messages dock icon
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.39 --ef y 0.97
|
|
sleep 2
|
|
|
|
# Click back to return home
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.1 --ef y 0.05
|
|
sleep 2
|
|
|
|
# Click Contacts dock icon
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.61 --ef y 0.97
|
|
sleep 2
|
|
|
|
# Click back to return home
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.1 --ef y 0.05
|
|
sleep 2
|
|
|
|
# Click Browser dock icon
|
|
adb shell am broadcast -a com.omixlab.mosis.INJECT_TOUCH \
|
|
--es touch_type "click" --ef x 0.84 --ef y 0.97
|
|
```
|
|
|
|
### Reading Logs
|
|
|
|
```bash
|
|
# Filter for Mosis logs
|
|
adb logcat -s MosisTest ServiceTester RMLUI
|
|
|
|
# Filter for navigation events
|
|
adb logcat -d | grep -iE "navigat|loaded|goBack|rml"
|
|
|
|
# Save to file
|
|
adb logcat -s MosisTest > mosis-log.txt
|
|
|
|
# Clear logs
|
|
adb logcat -c
|
|
```
|
|
|
|
### Expected Log Output
|
|
|
|
Successful navigation shows these log patterns:
|
|
```
|
|
RMLUI: navigateTo called with: dialer
|
|
Loading screen: apps/dialer/dialer.rml
|
|
RMLUI: Navigated to: dialer (history depth: 1)
|
|
|
|
RMLUI: goBack called (history depth: 1)
|
|
Loading screen: apps/home/home.rml
|
|
RMLUI: Back to: home
|
|
```
|
|
|
|
## Documentation Guidelines
|
|
|
|
**IMPORTANT**: Always document progress and new commands to avoid rediscovery.
|
|
|
|
### Where to Put Documentation
|
|
|
|
| Content Type | Location |
|
|
|--------------|----------|
|
|
| General concepts, architecture | `MosisService/CLAUDE.md` (this file) |
|
|
| Unreal plugin docs | `MosisUnreal/Plugins/MosisSDK/README.md` |
|
|
| Unity package docs | `MosisVR/Packages/com.omixlab.mosis_sdk/README.md` |
|
|
| Project-specific build commands | Each project's own docs |
|
|
|
|
**DO NOT** put documentation in the root `D:\Dev\Mosis\` directory - it is not versioned.
|
|
|
|
### What to Document
|
|
|
|
1. **Build commands** - Every new build command discovered or created
|
|
2. **Environment setup** - Required environment variables, SDK versions
|
|
3. **Architecture decisions** - Why something was done a certain way
|
|
4. **Issues and solutions** - Problems encountered and how they were fixed
|
|
5. **File locations** - Where important files are and what they do
|
|
|
|
### When to Document
|
|
|
|
- After completing a milestone or feature
|
|
- When discovering new build commands
|
|
- When fixing a non-obvious issue
|
|
- When adding new dependencies or requirements
|
|
|
|
## Game Engine Integrations
|
|
|
|
MosisService provides a virtual phone that game engines can display and interact with.
|
|
|
|
### Integration Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ MosisService │
|
|
│ (OpenGL ES rendering → AHardwareBuffer) │
|
|
└───────────────────────────┬─────────────────────────────────────┘
|
|
│ Binder IPC + Shared Memory
|
|
┌─────────────────┴─────────────────┐
|
|
▼ ▼
|
|
┌─────────────────────┐ ┌─────────────────────┐
|
|
│ MosisUnreal │ │ MosisVR │
|
|
│ (UE5.5 Plugin) │ │ (Unity Package) │
|
|
│ Vulkan Import │ │ Vulkan/OpenGL │
|
|
└─────────────────────┘ └─────────────────────┘
|
|
```
|
|
|
|
### MosisUnreal (Unreal Engine 5.5)
|
|
|
|
**Location**: `D:\Dev\Mosis\MosisUnreal\Plugins\MosisSDK\`
|
|
|
|
#### Build Commands
|
|
|
|
```batch
|
|
:: Windows Editor Build
|
|
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
|
|
MosisUnrealEditor Win64 Development ^
|
|
-Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"
|
|
|
|
:: Android APK Build
|
|
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
|
|
BuildCookRun ^
|
|
-project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
|
|
-platform=Android -clientconfig=Development ^
|
|
-build -cook -stage -pak -package -noP4
|
|
|
|
:: Clean Build (delete these folders first)
|
|
rmdir /s /q "Intermediate\Build"
|
|
rmdir /s /q "Binaries"
|
|
```
|
|
|
|
#### Output Files
|
|
|
|
| Build | Output |
|
|
|-------|--------|
|
|
| Windows Editor | `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll` |
|
|
| Android APK | `Binaries/Android/MosisUnreal-arm64.apk` |
|
|
| Android OBB | `Binaries/Android/main.1.com.omixlab.MosisUnreal.obb` |
|
|
|
|
#### Requirements
|
|
|
|
- Android SDK Platform 36 (for AIDL binder headers)
|
|
- Android Build Tools 36.1.0 (for AIDL compiler)
|
|
- `ANDROID_HOME` environment variable set
|
|
|
|
#### Quest Deployment
|
|
|
|
**IMPORTANT**: Git Bash on Windows converts Unix paths to Windows paths. Use `MSYS_NO_PATHCONV=1` prefix for all adb push commands.
|
|
|
|
```bash
|
|
# Set Quest device ID (check with: adb devices -l)
|
|
QUEST=2G0YC5ZF7W01T0
|
|
|
|
# Install APK
|
|
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk
|
|
|
|
# Create OBB temp directory
|
|
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"
|
|
|
|
# Push OBB to temp location (MUST use MSYS_NO_PATHCONV=1)
|
|
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/
|
|
|
|
# Move OBB to final location
|
|
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal"
|
|
adb -s $QUEST shell "mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"
|
|
|
|
# Start MosisService first, then MosisUnreal
|
|
adb -s $QUEST shell am start -n com.omixlab.mosis/.MainActivity
|
|
sleep 3
|
|
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
|
|
|
# Monitor logs
|
|
adb -s $QUEST logcat -s MosisSDK MosisOS MosisTest
|
|
```
|
|
|
|
**Full rebuild and deploy sequence**:
|
|
```bash
|
|
# 1. Build Android APK+OBB
|
|
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun \
|
|
-project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" \
|
|
-platform=Android -clientconfig=Development \
|
|
-build -cook -stage -pak -package -noP4
|
|
|
|
# 2. Stop app, push APK and OBB, restart
|
|
QUEST=2G0YC5ZF7W01T0
|
|
adb -s $QUEST shell am force-stop com.omixlab.MosisUnreal
|
|
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk
|
|
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"
|
|
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/
|
|
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal && mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"
|
|
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
|
```
|
|
|
|
**Common issues**:
|
|
- App stuck on loading screen: OBB not pushed or wrong version
|
|
- `adb push` path errors: Missing `MSYS_NO_PATHCONV=1` prefix
|
|
- Service not connecting: Start `com.omixlab.mosis` before the game
|
|
|
|
#### VR Pointer Interaction
|
|
|
|
The MosisSDK includes `UMosisPointerComponent` for VR ray-based touch interaction.
|
|
|
|
**Key Files**:
|
|
- `Public/MosisPointerComponent.h` - Component header
|
|
- `Private/MosisPointerComponent.cpp` - Implementation
|
|
|
|
**Features**:
|
|
- Raycast from component transform (attach as child of motion controller)
|
|
- Automatic detection of `AMosisPhoneActor` hits
|
|
- Touch state machine: Down/Move/Up events
|
|
- Optional Enhanced Input binding via `TriggerAction` property
|
|
- Manual control via `SetTriggerPressed(bool)`
|
|
- Debug ray visualization
|
|
|
|
**Blueprint Setup**:
|
|
1. Add `MosisPointerComponent` as child of `MotionControllerComponent`
|
|
2. Set `TriggerAction` to your trigger input action (optional)
|
|
3. Touch events are sent automatically when pointing at phone and triggering
|
|
|
|
**C++ Setup**:
|
|
```cpp
|
|
// In VR Pawn header
|
|
UPROPERTY(VisibleAnywhere)
|
|
UMosisPointerComponent* RightPointer;
|
|
|
|
// In constructor
|
|
RightPointer = CreateDefaultSubobject<UMosisPointerComponent>(TEXT("RightPointer"));
|
|
RightPointer->SetupAttachment(RightMotionController);
|
|
|
|
// In BeginPlay (if using Enhanced Input)
|
|
RightPointer->TriggerAction = TriggerInputAction;
|
|
```
|
|
|
|
**Manual Control (without Enhanced Input)**:
|
|
```cpp
|
|
// In your input handling code
|
|
void AMyVRPawn::OnTriggerPressed()
|
|
{
|
|
RightPointer->SetTriggerPressed(true);
|
|
}
|
|
|
|
void AMyVRPawn::OnTriggerReleased()
|
|
{
|
|
RightPointer->SetTriggerPressed(false);
|
|
}
|
|
```
|
|
|
|
**Component Properties**:
|
|
|
|
| Property | Type | Default | Description |
|
|
|----------|------|---------|-------------|
|
|
| `RayLength` | float | 500.0 | Maximum ray length in cm |
|
|
| `bShowDebugRay` | bool | false | Draw debug visualization |
|
|
| `DebugRayColor` | FLinearColor | Red | Ray color when not hitting |
|
|
| `DebugRayHitColor` | FLinearColor | Green | Ray color when hitting phone |
|
|
| `TraceChannel` | ECollisionChannel | Visibility | Collision channel for raycast |
|
|
| `TriggerAction` | UInputAction* | nullptr | Enhanced Input action for trigger |
|
|
|
|
**Blueprint Functions**:
|
|
|
|
| Function | Returns | Description |
|
|
|----------|---------|-------------|
|
|
| `IsPointingAtPhone()` | bool | True if ray hits a phone actor |
|
|
| `GetTargetPhone()` | AMosisPhoneActor* | Currently targeted phone (or nullptr) |
|
|
| `GetHitLocation()` | FVector | World location of ray hit |
|
|
| `GetRayOrigin()` | FVector | Ray start position |
|
|
| `GetRayDirection()` | FVector | Ray direction vector |
|
|
| `IsTriggerPressed()` | bool | Current trigger state |
|
|
| `SetTriggerPressed(bool)` | void | Manual trigger control |
|
|
|
|
**Touch State Machine**:
|
|
```
|
|
Idle ──[raycast hit]──► Hover ──[trigger]──► Pressed
|
|
▲ │ │
|
|
│ │ raycast miss │ trigger release
|
|
│ ▼ ▼
|
|
└──────────────────── Idle ◄────────────────────
|
|
|
|
Events:
|
|
- Idle → Pressed: SendTouch(Down)
|
|
- Pressed + moving: SendTouch(Move)
|
|
- Pressed → Idle: SendTouch(Up)
|
|
```
|
|
|
|
### MosisVR (Unity 6000.3.2f1)
|
|
|
|
**Location**: `D:\Dev\Mosis\MosisVR\Packages\com.omixlab.mosis_sdk\`
|
|
|
|
#### Direct APK Build (Recommended)
|
|
|
|
```batch
|
|
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
|
|
-batchmode -quit -nographics ^
|
|
-projectPath "D:\Dev\Mosis\MosisVR" ^
|
|
-executeMethod BuildScript.BuildAndroidDirectCI ^
|
|
-outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk"
|
|
```
|
|
|
|
#### Export + Gradle Build
|
|
|
|
For more control, export a Gradle project then build separately:
|
|
|
|
```batch
|
|
:: Step 1: Export from Unity
|
|
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
|
|
-batchmode -quit -nographics ^
|
|
-projectPath "D:\Dev\Mosis\MosisVR" ^
|
|
-executeMethod BuildScript.BuildAndroidCI ^
|
|
-export true ^
|
|
-outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR"
|
|
|
|
:: Step 2: Build with Gradle
|
|
cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
|
|
gradle assembleRelease
|
|
:: APK at: launcher\build\outputs\apk\release\launcher-release.apk
|
|
```
|
|
|
|
#### Unity Editor Manual Build
|
|
|
|
1. File > Build Settings > Android
|
|
2. Player Settings: IL2CPP, ARM64, Vulkan + OpenGLES3
|
|
3. For direct APK: Uncheck "Export Project", click Build
|
|
4. For export: Check "Export Project", click Export
|
|
|
|
#### Native Plugin Build (Manual)
|
|
|
|
The native plugin builds automatically via CMake during Unity's build. To rebuild manually:
|
|
|
|
```batch
|
|
cd Packages/com.omixlab.mosis_sdk/Plugins/Android/cpp
|
|
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
|
|
-DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29
|
|
cmake --build build
|
|
```
|
|
|
|
### Device Testing (Both Engines)
|
|
|
|
```bash
|
|
# Install MosisService first
|
|
adb install -r MosisService-debug.apk
|
|
|
|
# Install game client
|
|
adb install -r MosisUnreal-arm64.apk # or MosisVR.apk
|
|
|
|
# Launch service
|
|
adb shell am start -n com.omixlab.mosis/.MainActivity
|
|
|
|
# Launch client
|
|
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
|
# or for Unity:
|
|
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity
|
|
|
|
# Monitor all Mosis logs
|
|
adb logcat -s MosisSDK MosisTest RMLUI Vulkan
|
|
```
|
|
|
|
## Vulkan HardwareBuffer Import
|
|
|
|
Both game engines use Vulkan to import AHardwareBuffer from MosisService.
|
|
|
|
### Required Vulkan Extensions
|
|
|
|
```
|
|
VK_ANDROID_external_memory_android_hardware_buffer
|
|
VK_KHR_external_memory
|
|
VK_KHR_dedicated_allocation
|
|
```
|
|
|
|
### Import Pattern
|
|
|
|
```cpp
|
|
// 1. Query buffer properties
|
|
VkAndroidHardwareBufferPropertiesANDROID props = {
|
|
.sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID
|
|
};
|
|
VkAndroidHardwareBufferFormatPropertiesANDROID formatProps = {
|
|
.sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID
|
|
};
|
|
props.pNext = &formatProps;
|
|
vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &props);
|
|
|
|
// 2. Create image with external memory
|
|
VkExternalMemoryImageCreateInfo extInfo = {
|
|
.sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO,
|
|
.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID
|
|
};
|
|
|
|
AHardwareBuffer_Desc desc;
|
|
AHardwareBuffer_describe(buffer, &desc);
|
|
|
|
VkImageCreateInfo imageInfo = {
|
|
.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
|
|
.pNext = &extInfo,
|
|
.imageType = VK_IMAGE_TYPE_2D,
|
|
.format = formatProps.format,
|
|
.extent = {desc.width, desc.height, 1},
|
|
.mipLevels = 1,
|
|
.arrayLayers = 1,
|
|
.samples = VK_SAMPLE_COUNT_1_BIT,
|
|
.tiling = VK_IMAGE_TILING_OPTIMAL,
|
|
.usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT,
|
|
.sharingMode = VK_SHARING_MODE_EXCLUSIVE
|
|
};
|
|
vkCreateImage(device, &imageInfo, nullptr, &image);
|
|
|
|
// 3. Import memory from HardwareBuffer
|
|
VkImportAndroidHardwareBufferInfoANDROID importInfo = {
|
|
.sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID,
|
|
.buffer = buffer
|
|
};
|
|
|
|
VkMemoryDedicatedAllocateInfo dedicatedInfo = {
|
|
.sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO,
|
|
.pNext = &importInfo,
|
|
.image = image
|
|
};
|
|
|
|
VkMemoryAllocateInfo allocInfo = {
|
|
.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
|
|
.pNext = &dedicatedInfo,
|
|
.allocationSize = props.allocationSize,
|
|
.memoryTypeIndex = FindMemoryType(props.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)
|
|
};
|
|
vkAllocateMemory(device, &allocInfo, nullptr, &memory);
|
|
vkBindImageMemory(device, image, memory, 0);
|
|
```
|
|
|
|
### Synchronization
|
|
|
|
Currently using CPU synchronization (`vkQueueWaitIdle` / `glFinish`). Future improvement: use Vulkan semaphores for GPU-GPU sync.
|
|
|
|
### Double Buffering
|
|
|
|
The imported image is copied to a local texture each frame to prevent data races with MosisService rendering.
|
|
|
|
## Developer Portal
|
|
|
|
The Developer Portal is a web application for app developers to publish and manage their Mosis apps. Planning documents are in `DEV_PORTAL_M01-M12.md` files.
|
|
|
|
### Architecture Decisions
|
|
|
|
| Component | Technology | Rationale |
|
|
|-----------|------------|-----------|
|
|
| Backend | Go 1.22+ | Simple, fast, single binary deployment |
|
|
| Router | Chi | Lightweight, idiomatic Go HTTP routing |
|
|
| Database | SQLite + Litestream | Zero-ops, continuous backup to NAS storage |
|
|
| Frontend | htmx + Go templates | Server-rendered, minimal JS, fast |
|
|
| Storage | Synology NAS filesystem | Self-hosted, local volume mounts |
|
|
| Auth | OAuth2 (GitHub/Google) + JWT | golang-jwt for tokens, Ed25519 for signing |
|
|
| CLI | Go + Cobra | Cross-platform single binary |
|
|
| Docs | Hugo + Docsy | Static site served from /docs/ |
|
|
|
|
### Deployment Target
|
|
|
|
Self-hosted on Synology NAS via Docker:
|
|
|
|
```
|
|
/volume1/mosis/
|
|
├── data/
|
|
│ ├── portal.db # Main SQLite database
|
|
│ └── telemetry.db # Separate telemetry database
|
|
├── packages/ # Uploaded app packages
|
|
│ └── {dev_id}/{app_id}/{version}/
|
|
├── backups/ # Litestream replicas
|
|
└── docs/ # Hugo static site output
|
|
```
|
|
|
|
### Milestone Documents
|
|
|
|
| File | Topic | Status |
|
|
|------|-------|--------|
|
|
| DEV_PORTAL_M01_OVERVIEW.md | Project overview | Decided |
|
|
| DEV_PORTAL_M02_WEB_STACK.md | Go + Chi + htmx | Decided |
|
|
| DEV_PORTAL_M03_DATABASE.md | SQLite + Litestream | Decided |
|
|
| DEV_PORTAL_M04_AUTH.md | OAuth2 + JWT + Ed25519 | Decided |
|
|
| DEV_PORTAL_M05_FRONTEND.md | htmx + Go templates | Decided |
|
|
| DEV_PORTAL_M06_API.md | REST API design | Decided |
|
|
| DEV_PORTAL_M07_STORAGE.md | NAS filesystem | Decided |
|
|
| DEV_PORTAL_M08_TELEMETRY.md | SQLite + background workers | Decided |
|
|
| DEV_PORTAL_M09_REVIEW.md | Go validation workers | Decided |
|
|
| DEV_PORTAL_M10_DEVICE.md | C++ AppManager | Decided |
|
|
| DEV_PORTAL_M11_CLI.md | Go + Cobra CLI | Decided |
|
|
| DEV_PORTAL_M12_DOCS.md | Hugo + Docsy | Decided |
|
|
|
|
### Key Design Principles
|
|
|
|
1. **Single container** - Portal runs as one Docker container on NAS
|
|
2. **No external services** - SQLite, local filesystem, Pagefind search
|
|
3. **Pure Go** - `modernc.org/sqlite` (no CGO required)
|
|
4. **Server-rendered** - htmx for interactivity, no SPA framework
|
|
5. **Background workers** - Go goroutines for aggregation/cleanup jobs
|