464 lines
16 KiB
Markdown
464 lines
16 KiB
Markdown
# Mosis Virtual Smartphone Platform - Roadmap
|
|
|
|
## Vision
|
|
|
|
Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality.
|
|
|
|
### Key Use Cases
|
|
|
|
1. **In-Game Phone**: Players use a virtual phone in VR games for communication, utilities, entertainment
|
|
2. **Cross-Game Communication**: Phone identity persists across different games (unified contacts, messages)
|
|
3. **Real-World Bridge**: Connect to real smartphones via WebRTC for mixed-reality communication
|
|
4. **Virtual Hardware**: Game engine provides camera textures, audio, and other "hardware" to the phone OS
|
|
5. **Store Ecosystem**: Downloadable apps from a store, self-contained packages with isolation
|
|
|
|
### Integration Points
|
|
|
|
| Platform | Location | Status |
|
|
|----------|----------|--------|
|
|
| Unity Plugin | `D:\Dev\Mosis\Mosis Unity` | Binder client, raycast, viewport |
|
|
| Unreal Plugin | `D:\Dev\Mosis\Mosis Unreal` | WIP |
|
|
| Desktop Designer | `MosisService/designer/` | Complete |
|
|
| Designer Tests | `MosisService/designer-test/` | Mostly complete |
|
|
|
|
---
|
|
|
|
## Milestones Overview
|
|
|
|
| # | Milestone | Status | Description |
|
|
|---|-----------|--------|-------------|
|
|
| 1 | Cross-Platform Kernel | ✅ Complete | Desktop designer with shared kernel code |
|
|
| 2 | Testing Framework | ✅ Complete | Automated UI testing and inspection |
|
|
| 3 | Virtual Hardware | ❌ Not started | Camera, mic, speaker, filesystem APIs |
|
|
| 4 | App Sandboxing | ❌ Not started | Lua/WASM runtime, package format |
|
|
| 5 | WebRTC Bridge | ❌ Not started | Phone-to-phone communication |
|
|
| 6 | System Apps | 🔶 75% | Core phone apps |
|
|
| 7 | Game Integration | ❌ Not started | Unity/Unreal plugin polish |
|
|
|
|
---
|
|
|
|
## Milestone 1: Cross-Platform Kernel ✅ COMPLETE
|
|
|
|
**Goal**: Desktop designer with shared kernel code for rapid UI iteration.
|
|
|
|
### Completed Tasks
|
|
|
|
- [x] Platform abstraction layer (`src/main/kernel/include/`)
|
|
- `platform.h` - IPlatform, IGraphicsContext, IRenderTarget
|
|
- `service_interface.h` - IKernel, IServiceListener
|
|
- `file_interface.h` - IFileInterface extending Rml::FileInterface
|
|
- [x] Desktop designer project (`designer/`)
|
|
- GLFW + OpenGL 3.3 context
|
|
- Hot-reload on RML/RCSS/Lua changes
|
|
- Mouse input maps to touch events
|
|
- [x] UI designs imported from MosisDesigner
|
|
- Theme and component stylesheets
|
|
- Navigation system (Lua)
|
|
- Fonts and icons
|
|
- [x] Android build unchanged and working
|
|
|
|
---
|
|
|
|
## Milestone 2: Testing Automation ✅ COMPLETE
|
|
|
|
**Goal**: Automated UI testing for rapid iteration and AI agent verification.
|
|
|
|
### Completed Tasks
|
|
|
|
- [x] UI hierarchy dumping to JSON
|
|
- [x] PNG screenshot capture (F12 key)
|
|
- [x] Agent-compatible JSON test results
|
|
- [x] `designer-test` executable
|
|
- WindowController (Windows SendInput API)
|
|
- HierarchyReader (element lookup by ID/class)
|
|
- LogParser (navigation event verification)
|
|
- [x] All 5 navigation tests passing
|
|
- [x] Android event injection via ADB broadcast
|
|
- [x] **Action Recording** - Capture tap, swipe, long_press to JSON with timestamps
|
|
- [x] **Action Playback** - Load recorded JSON and replay with timing
|
|
- [x] **Screenshot Diff** - Pixel-level PNG comparison with configurable tolerance
|
|
- [x] **GLFW Input Hooks** - Automatic mouse/key recording via forked backend
|
|
|
|
### Test Recording Format
|
|
|
|
```json
|
|
{
|
|
"name": "Navigate to contacts and back",
|
|
"recorded_at": "2024-01-16T12:00:00Z",
|
|
"resolution": {"width": 677, "height": 1202},
|
|
"actions": [
|
|
{"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
|
|
{"type": "wait", "duration": 1000, "timestamp": 100},
|
|
{"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
|
|
{"type": "wait", "duration": 500, "timestamp": 1200}
|
|
],
|
|
"assertions": [
|
|
{"after_action": 0, "type": "screen_is", "expected": "contacts"},
|
|
{"after_action": 2, "type": "screen_is", "expected": "home"}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Milestone 3: Virtual Hardware ❌ NOT STARTED
|
|
|
|
**Goal**: Hardware-like APIs backed by game engine or real devices.
|
|
|
|
### 3.1 Camera Interface
|
|
|
|
```cpp
|
|
class ICamera {
|
|
virtual void RequestFrame(FrameCallback callback) = 0;
|
|
virtual void SetResolution(int width, int height) = 0;
|
|
virtual bool IsAvailable() const = 0;
|
|
};
|
|
```
|
|
|
|
**Implementations**:
|
|
- **Game Mode**: Receives texture from Unity/Unreal
|
|
- **Desktop Mode**: System webcam (optional)
|
|
- **Android Test Mode**: SharedTexture from MainActivity
|
|
- **Mock Mode**: Test patterns
|
|
|
|
### 3.2 Microphone Interface
|
|
|
|
```cpp
|
|
class IMicrophone {
|
|
virtual void StartCapture(AudioCallback callback) = 0;
|
|
virtual void StopCapture() = 0;
|
|
virtual bool IsAvailable() const = 0;
|
|
};
|
|
```
|
|
|
|
### 3.3 Speaker Interface
|
|
|
|
```cpp
|
|
class ISpeaker {
|
|
virtual void PlayAudio(AudioBuffer buffer) = 0;
|
|
virtual void SetVolume(float volume) = 0;
|
|
};
|
|
```
|
|
|
|
### 3.4 Filesystem Interface
|
|
|
|
```cpp
|
|
class IFileSystem {
|
|
virtual FileHandle Open(const std::string& path, Mode mode) = 0;
|
|
virtual std::vector<std::string> List(const std::string& dir) = 0;
|
|
virtual bool CreateDirectory(const std::string& path) = 0;
|
|
virtual bool Delete(const std::string& path) = 0;
|
|
};
|
|
```
|
|
|
|
### 3.5 Network Interface
|
|
|
|
```cpp
|
|
class INetwork {
|
|
virtual HttpResponse Fetch(const HttpRequest& request) = 0;
|
|
virtual WebSocket Connect(const std::string& url) = 0;
|
|
virtual bool IsOnline() const = 0;
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Milestone 4: App Sandboxing ❌ NOT STARTED
|
|
|
|
**Goal**: Secure app runtime with defined package format.
|
|
|
|
### 4.1 Runtime Decision
|
|
|
|
| Aspect | Lua | WASM |
|
|
|--------|-----|------|
|
|
| Isolation | Weak | Strong |
|
|
| Performance | Good for UI | Near-native |
|
|
| RmlUi Integration | Native | Needs bridge |
|
|
| Ecosystem | Small | Large |
|
|
|
|
**Recommendation**: Hybrid approach
|
|
- Lua for UI scripting (RmlUi integration)
|
|
- WASM for app logic needing isolation (future)
|
|
|
|
### 4.2 Package Format (.mpkg)
|
|
|
|
```
|
|
myapp.mpkg/
|
|
├── manifest.json # Metadata, permissions, entry
|
|
├── ui/
|
|
│ ├── main.rml
|
|
│ ├── styles.rcss
|
|
│ └── scripts/
|
|
├── assets/
|
|
│ ├── icons/
|
|
│ └── images/
|
|
└── data/
|
|
```
|
|
|
|
### 4.3 Manifest Schema
|
|
|
|
```json
|
|
{
|
|
"id": "com.example.myapp",
|
|
"name": "My App",
|
|
"version": "1.0.0",
|
|
"permissions": ["camera", "network", "storage"],
|
|
"entry": "ui/main.rml",
|
|
"icon": "assets/icons/app.png"
|
|
}
|
|
```
|
|
|
|
### 4.4 App Lifecycle
|
|
|
|
```
|
|
INSTALLED → LAUNCHING → RUNNING → PAUSED → STOPPED → UNINSTALLED
|
|
```
|
|
|
|
---
|
|
|
|
## Milestone 5: WebRTC Bridge ❌ NOT STARTED
|
|
|
|
**Goal**: Cross-device communication via WebRTC.
|
|
|
|
### 5.1 Dependencies
|
|
|
|
- libdatachannel (add to vcpkg)
|
|
|
|
### 5.2 Communication Protocol
|
|
|
|
```json
|
|
{
|
|
"type": "call" | "message" | "file" | "screen_share",
|
|
"from": "phone_id",
|
|
"to": "phone_id",
|
|
"payload": { ... }
|
|
}
|
|
```
|
|
|
|
### 5.3 Use Cases
|
|
|
|
- Voice/video calls between virtual phones
|
|
- Text messaging across games
|
|
- File sharing
|
|
- Screen sharing
|
|
|
|
### 5.4 Components
|
|
|
|
- **WebRTCBridge** class for data channels
|
|
- **Signaling Server** (WebSocket-based)
|
|
- **Real Smartphone Bridge** (companion app)
|
|
|
|
---
|
|
|
|
## Milestone 6: System Apps 🔶 75% COMPLETE
|
|
|
|
### Completed Apps
|
|
|
|
| App | Features | Status |
|
|
|-----|----------|--------|
|
|
| Home | App grid, dock, navigation | ✅ Complete |
|
|
| Dialer | Keypad, call UI (mock) | ✅ Complete |
|
|
| Messages | Conversation list, chat view | ✅ Complete |
|
|
| Contacts | Contact list, search, detail | ✅ Complete |
|
|
| Settings | Display, sound, about | ✅ Complete |
|
|
| Browser | URL bar, placeholder | ✅ Complete |
|
|
|
|
### Remaining Apps
|
|
|
|
| App | Features | Status |
|
|
|-----|----------|--------|
|
|
| Store | Browse apps, install packages | ❌ UI only |
|
|
| Camera | Viewfinder, capture, gallery | ❌ UI only |
|
|
| Music/Audio | Playback, playlists | ❌ Outline only |
|
|
|
|
### App Data Persistence (Future)
|
|
|
|
- Contacts: JSON or SQLite storage
|
|
- Messages: Conversation history
|
|
- Settings: Preferences file
|
|
- Photos: Virtual filesystem
|
|
|
|
---
|
|
|
|
## Milestone 7: Game Integration ❌ NOT STARTED
|
|
|
|
**Goal**: Production-ready game engine plugins.
|
|
|
|
### 7.1 Unity Package
|
|
|
|
- [ ] Improved raycast interaction
|
|
- [ ] Hardware button simulation (volume, power)
|
|
- [ ] Virtual camera provider (RenderTexture → phone)
|
|
- [ ] Virtual microphone provider
|
|
- [ ] Virtual speaker provider
|
|
|
|
### 7.2 Unreal Plugin
|
|
|
|
- [ ] Full Binder client implementation
|
|
- [ ] Blueprint integration
|
|
- [ ] Same features as Unity
|
|
|
|
### 7.3 Documentation
|
|
|
|
- [ ] How to embed phone in game scene
|
|
- [ ] How to provide virtual hardware
|
|
- [ ] How to handle phone events
|
|
|
|
---
|
|
|
|
## Extended Features (Future)
|
|
|
|
| Feature | Description |
|
|
|---------|-------------|
|
|
| Notifications | System notification center, per-app channels |
|
|
| Widgets | Home screen widgets, live data |
|
|
| Gestures | Swipe up for home, swipe down for notifications |
|
|
| Themes | Dark/light mode, custom colors |
|
|
| Accessibility | Font size, high contrast, TTS |
|
|
| Multi-Window | Split-screen, picture-in-picture |
|
|
| Clipboard | Copy/paste between apps and game |
|
|
| Voice Assistant | Wake word, voice commands |
|
|
| Virtual Location | GPS from game, map integration |
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ VR Game (Unity/Unreal) │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │
|
|
│ │ Phone Render │ │ Touch/Button │ │ Virtual Hardware Provider │ │
|
|
│ │ Viewport │ │ Input │ │ (Camera, Mic, Speakers) │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └────────────┬─────────────┘ │
|
|
└─────────┼─────────────────┼────────────────────────┼────────────────┘
|
|
│ Binder IPC │ Binder IPC │ Binder IPC
|
|
▼ ▼ ▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Mosis Service │
|
|
│ ┌─────────────────────────────────────────────────────────────┐ │
|
|
│ │ Kernel │ │
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
|
|
│ │ │ RmlUi │ │ App Runtime │ │ Virtual Hardware │ │ │
|
|
│ │ │ Renderer │ │ (Lua) │ │ Abstraction │ │ │
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
|
|
│ │ │ App │ │ IPC │ │ libdatachannel │ │ │
|
|
│ │ │ Manager │ │ Router │ │ (WebRTC Bridge) │ │ │
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
|
|
│ └─────────────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
│
|
|
│ WebRTC (libdatachannel)
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Real Smartphone / Other Devices │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Folder Structure
|
|
|
|
```
|
|
MosisService/
|
|
├── src/main/
|
|
│ ├── kernel/ # Shared cross-platform kernel
|
|
│ │ ├── include/ # Platform abstractions
|
|
│ │ └── src/ # Shared implementation
|
|
│ ├── cpp/ # Android-specific native code
|
|
│ ├── java/ # Kotlin/Java code
|
|
│ └── assets/ # Shared UI assets
|
|
│ ├── apps/ # System apps
|
|
│ │ ├── home/
|
|
│ │ ├── dialer/
|
|
│ │ ├── messages/
|
|
│ │ ├── contacts/
|
|
│ │ ├── settings/
|
|
│ │ ├── browser/
|
|
│ │ ├── store/ # TODO
|
|
│ │ └── camera/ # TODO
|
|
│ ├── ui/ # Shared stylesheets
|
|
│ ├── scripts/ # Lua scripts
|
|
│ ├── icons/ # TGA icons
|
|
│ └── fonts/ # TTF fonts
|
|
│
|
|
├── designer/ # Desktop designer
|
|
│ ├── src/
|
|
│ │ ├── testing/ # UI inspection, capture
|
|
│ │ └── ...
|
|
│ └── build/
|
|
│
|
|
├── designer-test/ # Automated UI tests
|
|
│ ├── src/
|
|
│ └── build/
|
|
│
|
|
└── docs/ # Documentation (future)
|
|
```
|
|
|
|
---
|
|
|
|
## Build Commands
|
|
|
|
```bash
|
|
# Android
|
|
./gradlew assembleDebug
|
|
./gradlew installDebug
|
|
|
|
# Desktop Designer
|
|
cd designer
|
|
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
|
|
cmake --build build --config Debug
|
|
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml
|
|
|
|
# Designer Tests
|
|
cd designer-test
|
|
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
|
|
cmake --build build --config Debug
|
|
./build/Debug/designer-test.exe
|
|
```
|
|
|
|
---
|
|
|
|
## Current Sprint: Complete Partial Tasks
|
|
|
|
### Priority 1: Testing Framework Completion ✅ DONE
|
|
- [x] Action recording (capture interactions to JSON)
|
|
- [x] Action playback (replay with timing)
|
|
- [x] Screenshot diff (visual regression)
|
|
- [x] GLFW input hooks for automatic recording
|
|
|
|
### Priority 2: Remaining System Apps
|
|
- [ ] Store app (UI only - browse, install)
|
|
- [ ] Camera app (UI + shared texture from MainActivity)
|
|
- [ ] Music app (UI outline only)
|
|
|
|
### Priority 3: App Data Persistence
|
|
- [ ] JSON/SQLite storage layer
|
|
- [ ] Contact CRUD operations
|
|
- [ ] Message history
|
|
- [ ] Settings persistence
|
|
|
|
---
|
|
|
|
## Resources
|
|
|
|
### Material Design
|
|
|
|
- **Icons**: `D:\Dev\Mosis\MosisDesigner\material-design-icons`
|
|
- 20 categories, SVG/PNG/Font formats
|
|
- Browse at https://fonts.google.com/icons
|
|
|
|
- **Components**: `D:\Dev\Mosis\MosisDesigner\material-design-lite`
|
|
- Reference implementation for design patterns
|
|
|
|
### Documentation
|
|
|
|
- `CLAUDE.md` - Development guide
|
|
- `TESTING.md` - Testing framework documentation
|
|
- `ROADMAP.md` - This file
|
|
|
|
---
|
|
|
|
*Last updated: 2026-01-16*
|