omar/MosisService
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
341d663b366441682b49523ac67dbe2df8845183
MosisService/docs/ROADMAP.md

393 lines
15 KiB
Markdown
Raw Blame History

# Mosis Virtual Smartphone Platform - Roadmap
## Vision
Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality.
### Key Use Cases
1. **In-Game Phone**: Players use a virtual phone in VR games for communication, utilities, entertainment
2. **Cross-Game Communication**: Phone identity persists across different games (unified contacts, messages)
3. **Real-World Bridge**: Connect to real smartphones via WebRTC for mixed-reality communication
4. **Virtual Hardware**: Game engine provides camera textures, audio, and other "hardware" to the phone OS
5. **Store Ecosystem**: Downloadable apps from a store, self-contained packages with isolation
### Integration Points
| Platform | Location | Status |
|----------|----------|--------|
| Unity Plugin | `D:\Dev\Mosis\Mosis Unity` | Binder client, raycast, viewport |
| Unreal Plugin | `D:\Dev\Mosis\Mosis Unreal` | WIP |
| Desktop Designer | `MosisService/designer/` | Complete |
| Designer Tests | `MosisService/designer-test/` | Mostly complete |
---
## Milestones Overview
| # | Milestone | Status | Description |
|---|-----------|--------|-------------|
| 1 | Cross-Platform Kernel | ✅ Complete | Desktop designer with shared kernel code |
| 2 | Testing Framework | ✅ Complete | Automated UI testing and inspection |
| 3 | Virtual Hardware | ✅ Complete | Camera, mic, speaker, filesystem APIs (via sandbox) |
| 4 | App Sandboxing | ✅ Complete | Lua runtime, package format, 149 security tests |
| 5 | WebRTC Bridge | ❌ Not started | Phone-to-phone communication |
| 6 | System Apps | 🔶 75% | Core phone apps |
| 7 | Game Integration | 🔶 In Progress | Unity/Unreal plugins (Vulkan working) |
---
## Milestone 1: Cross-Platform Kernel ✅ COMPLETE
**Goal**: Desktop designer with shared kernel code for rapid UI iteration.
### Completed Tasks
- [x] Platform abstraction layer (`src/main/kernel/include/`)
- `platform.h` - IPlatform, IGraphicsContext, IRenderTarget
- `service_interface.h` - IKernel, IServiceListener
- `file_interface.h` - IFileInterface extending Rml::FileInterface
- [x] Desktop designer project (`designer/`)
- GLFW + OpenGL 3.3 context
- Hot-reload on RML/RCSS/Lua changes
- Mouse input maps to touch events
- [x] UI designs imported from MosisDesigner
- Theme and component stylesheets
- Navigation system (Lua)
- Fonts and icons
- [x] Android build unchanged and working
---
## Milestone 2: Testing Automation ✅ COMPLETE
**Goal**: Automated UI testing for rapid iteration and AI agent verification.
### Completed Tasks
- [x] UI hierarchy dumping to JSON
- [x] PNG screenshot capture (F12 key)
- [x] Agent-compatible JSON test results
- [x] `designer-test` executable
- WindowController (Windows SendInput API)
- HierarchyReader (element lookup by ID/class)
- LogParser (navigation event verification)
- [x] All 5 navigation tests passing
- [x] Android event injection via ADB broadcast
- [x] **Action Recording** - Capture tap, swipe, long_press to JSON with timestamps
- [x] **Action Playback** - Load recorded JSON and replay with timing
- [x] **Screenshot Diff** - Pixel-level PNG comparison with configurable tolerance
- [x] **GLFW Input Hooks** - Automatic mouse/key recording via forked backend
### Test Recording Format
```json
{
"name": "Navigate to contacts and back",
"recorded_at": "2024-01-16T12:00:00Z",
"resolution": {"width": 677, "height": 1202},
"actions": [
{"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
{"type": "wait", "duration": 1000, "timestamp": 100},
{"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
{"type": "wait", "duration": 500, "timestamp": 1200}
],
"assertions": [
{"after_action": 0, "type": "screen_is", "expected": "contacts"},
{"after_action": 2, "type": "screen_is", "expected": "home"}
]
}
```
---
## Milestone 3: Virtual Hardware ✅ COMPLETE
**Goal**: Hardware-like APIs backed by game engine or real devices.
Implemented as part of the Lua Sandbox system. See [SANDBOX_MILESTONES.md](SANDBOX_MILESTONES.md) for details.
### Completed Components
| Component | Sandbox Milestone | Location |
|-----------|------------------|----------|
| Camera | Milestone 11 | `sandbox/camera_interface.cpp` |
| Microphone | Milestone 12 | `sandbox/microphone_interface.cpp` |
| Audio Output | Milestone 13 | `sandbox/audio_output.cpp` |
| Location/GPS | Milestone 14 | `sandbox/location_interface.cpp` |
| Sensors | Milestone 15 | `sandbox/sensor_interface.cpp` |
| Bluetooth | Milestone 16 | `sandbox/bluetooth_interface.cpp` |
| Filesystem | Milestone 7 | `sandbox/virtual_fs.cpp` |
| Network/HTTP | Milestone 9 | `sandbox/network_manager.cpp` |
| WebSocket | Milestone 10 | `sandbox/websocket_manager.cpp` |
---
## Milestone 4: App Sandboxing ✅ COMPLETE
**Goal**: Secure app runtime with defined package format.
Implemented with 20 sub-milestones covering security, APIs, and hardware interfaces. 149 security tests passing.
See:
- [SANDBOX_MILESTONES.md](SANDBOX_MILESTONES.md) - Implementation milestones
- [LUA-SANDBOX.md](LUA-SANDBOX.md) - Security features, permissions, APIs
- [APP-MANAGEMENT.md](APP-MANAGEMENT.md) - Package format, manifest, lifecycle
### Key Features
- **Lua Runtime**: Isolated `lua_State` per app with dangerous globals removed
- **Resource Limits**: Memory (16MB), CPU (instruction hooks), timers (100 max)
- **Permission System**: Normal/Dangerous/Signature categories with user gesture requirements
- **Virtual Filesystem**: Per-app storage with 50MB quota
- **SQLite Database**: Per-app with authorizer blocking dangerous operations
- **Network Security**: HTTPS required, private IP blocking, domain whitelists
- **Hardware Interfaces**: Camera, mic, location, sensors with mandatory indicators
---
## Milestone 5: WebRTC Bridge ❌ NOT STARTED
**Goal**: Cross-device communication via WebRTC.
### 5.1 Dependencies
- libdatachannel (add to vcpkg)
### 5.2 Communication Protocol
```json
{
"type": "call" | "message" | "file" | "screen_share",
"from": "phone_id",
"to": "phone_id",
"payload": { ... }
}
```
### 5.3 Use Cases
- Voice/video calls between virtual phones
- Text messaging across games
- File sharing
- Screen sharing
### 5.4 Components
- **WebRTCBridge** class for data channels
- **Signaling Server** (WebSocket-based)
- **Real Smartphone Bridge** (companion app)
---
## Milestone 6: System Apps 🔶 75% COMPLETE
### Completed Apps
| App | Features | Status |
|-----|----------|--------|
| Home | App grid, dock, navigation | ✅ Complete |
| Dialer | Keypad, call UI (mock) | ✅ Complete |
| Messages | Conversation list, chat view | ✅ Complete |
| Contacts | Contact list, search, detail | ✅ Complete |
| Settings | Display, sound, about | ✅ Complete |
| Browser | URL bar, placeholder | ✅ Complete |
### Remaining Apps
| App | Features | Status |
|-----|----------|--------|
| Store | Browse apps, install packages | ❌ UI only |
| Camera | Viewfinder, capture, gallery | ❌ UI only |
| Music/Audio | Playback, playlists | ❌ Outline only |
### App Data Persistence (Future)
- Contacts: JSON or SQLite storage
- Messages: Conversation history
- Settings: Preferences file
- Photos: Virtual filesystem
---
## Milestone 7: Game Integration ❌ NOT STARTED
**Goal**: Production-ready game engine plugins.
### 7.1 Unity Package
- [ ] Improved raycast interaction
- [ ] Hardware button simulation (volume, power)
- [ ] Virtual camera provider (RenderTexture → phone)
- [ ] Virtual microphone provider
- [ ] Virtual speaker provider
### 7.2 Unreal Plugin
- [ ] Full Binder client implementation
- [ ] Blueprint integration
- [ ] Same features as Unity
### 7.3 Documentation
- [ ] How to embed phone in game scene
- [ ] How to provide virtual hardware
- [ ] How to handle phone events
---
## Extended Features (Future)
| Feature | Description |
|---------|-------------|
| Notifications | System notification center, per-app channels |
| Widgets | Home screen widgets, live data |
| Gestures | Swipe up for home, swipe down for notifications |
| Themes | Dark/light mode, custom colors |
| Accessibility | Font size, high contrast, TTS |
| Multi-Window | Split-screen, picture-in-picture |
| Clipboard | Copy/paste between apps and game |
| Voice Assistant | Wake word, voice commands |
| Virtual Location | GPS from game, map integration |
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ VR Game (Unity/Unreal) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │
│ │ Phone Render │ │ Touch/Button │ │ Virtual Hardware Provider │ │
│ │ Viewport │ │ Input │ │ (Camera, Mic, Speakers) │ │
│ └──────┬───────┘ └──────┬───────┘ └────────────┬─────────────┘ │
└─────────┼─────────────────┼────────────────────────┼────────────────┘
│ Binder IPC │ Binder IPC │ Binder IPC
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ Mosis Service │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Kernel │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
│ │ │ RmlUi │ │ App Runtime │ │ Virtual Hardware │ │ │
│ │ │ Renderer │ │ (Lua) │ │ Abstraction │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
│ │ │ App │ │ IPC │ │ libdatachannel │ │ │
│ │ │ Manager │ │ Router │ │ (WebRTC Bridge) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
│ WebRTC (libdatachannel)
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Real Smartphone / Other Devices │
└─────────────────────────────────────────────────────────────────────┘
```
---
## Folder Structure
```
MosisService/
├── src/main/
│ ├── kernel/ # Shared cross-platform kernel
│ │ ├── include/ # Platform abstractions
│ │ └── src/ # Shared implementation
│ ├── cpp/ # Android-specific native code
│ ├── java/ # Kotlin/Java code
│ └── assets/ # Shared UI assets
│ ├── apps/ # System apps
│ │ ├── home/
│ │ ├── dialer/
│ │ ├── messages/
│ │ ├── contacts/
│ │ ├── settings/
│ │ ├── browser/
│ │ ├── store/ # TODO
│ │ └── camera/ # TODO
│ ├── ui/ # Shared stylesheets
│ ├── scripts/ # Lua scripts
│ ├── icons/ # TGA icons
│ └── fonts/ # TTF fonts
│
├── designer/ # Desktop designer
│ ├── src/
│ │ ├── testing/ # UI inspection, capture
│ │ └── ...
│ └── build/
│
├── designer-test/ # Automated UI tests
│ ├── src/
│ └── build/
│
└── docs/ # Documentation (future)
```
---
## Build Commands
```bash
# Android
./gradlew assembleDebug
./gradlew installDebug
# Desktop Designer
cd designer
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml
# Designer Tests
cd designer-test
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/designer-test.exe
```
---
## Current Sprint: Next Steps
### Priority 1: System Apps Completion
- [ ] Store app (UI only - browse, install)
- [ ] Camera app (UI + game engine texture)
- [ ] Music app (UI outline only)
### Priority 2: Game Engine Integration
- [x] MosisUnreal - Vulkan texture import working
- [ ] MosisUnreal - Blueprint API polish
- [ ] MosisVR (Unity) - Vulkan backend
### Priority 3: WebRTC Bridge (Milestone 5)
- [ ] libdatachannel integration
- [ ] Phone-to-phone communication
- [ ] Real smartphone bridge app
---
## Resources
### Material Design
- **Icons**: `D:\Dev\Mosis\MosisDesigner\material-design-icons`
- 20 categories, SVG/PNG/Font formats
- Browse at https://fonts.google.com/icons
- **Components**: `D:\Dev\Mosis\MosisDesigner\material-design-lite`
- Reference implementation for design patterns
### Documentation
All documentation is in `docs/`:
- `CLAUDE.md` - Development guide
- `SANDBOX_MILESTONES.md` - Sandbox implementation details
- `ROADMAP.md` - This file
---
*Last updated: 2026-01-19*
Reference in New Issue View Git Blame Copy Permalink