# Mosis Virtual Smartphone Platform - Roadmap

## Vision

Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality.

### Key Use Cases

1. **In-Game Phone**: Players use a virtual phone in VR games for communication, utilities, entertainment
2. **Cross-Game Communication**: Phone identity persists across different games (unified contacts, messages)
3. **Real-World Bridge**: Connect to real smartphones via WebRTC for mixed-reality communication
4. **Virtual Hardware**: Game engine provides camera textures, audio, and other "hardware" to the phone OS
5. **Store Ecosystem**: Downloadable apps from a store, self-contained packages with isolation

### Integration Points

| Platform | Location | Status |
|----------|----------|--------|
| Unity Plugin | `D:\Dev\Mosis\Mosis Unity` | Binder client, raycast, viewport |
| Unreal Plugin | `D:\Dev\Mosis\Mosis Unreal` | WIP |
| Desktop Designer | `MosisService/designer/` | Complete |
| Designer Tests | `MosisService/designer-test/` | Mostly complete |

---

## Milestones Overview

| # | Milestone | Status | Description |
|---|-----------|--------|-------------|
| 1 | Cross-Platform Kernel | ✅ Complete | Desktop designer with shared kernel code |
| 2 | Testing Framework | ✅ Complete | Automated UI testing and inspection |
| 3 | Virtual Hardware | ✅ Complete | Camera, mic, speaker, filesystem APIs (via sandbox) |
| 4 | App Sandboxing | ✅ Complete | Lua runtime, package format, 149 security tests |
| 5 | WebRTC Bridge | ❌ Not started | Phone-to-phone communication |
| 6 | System Apps | 🔶 75% | Core phone apps |
| 7 | Game Integration | 🔶 In Progress | Unity/Unreal plugins (Vulkan working) |

---

## Milestone 1: Cross-Platform Kernel ✅ COMPLETE

**Goal**: Desktop designer with shared kernel code for rapid UI iteration.

### Completed Tasks

- [x] Platform abstraction layer (`src/main/kernel/include/`)
  - `platform.h` - IPlatform, IGraphicsContext, IRenderTarget
  - `service_interface.h` - IKernel, IServiceListener
  - `file_interface.h` - IFileInterface extending Rml::FileInterface
- [x] Desktop designer project (`designer/`)
  - GLFW + OpenGL 3.3 context
  - Hot-reload on RML/RCSS/Lua changes
  - Mouse input maps to touch events
- [x] UI designs imported from MosisDesigner
  - Theme and component stylesheets
  - Navigation system (Lua)
  - Fonts and icons
- [x] Android build unchanged and working

---

## Milestone 2: Testing Automation ✅ COMPLETE

**Goal**: Automated UI testing for rapid iteration and AI agent verification.

### Completed Tasks

- [x] UI hierarchy dumping to JSON
- [x] PNG screenshot capture (F12 key)
- [x] Agent-compatible JSON test results
- [x] `designer-test` executable
  - WindowController (Windows SendInput API)
  - HierarchyReader (element lookup by ID/class)
  - LogParser (navigation event verification)
- [x] All 5 navigation tests passing
- [x] Android event injection via ADB broadcast
- [x] **Action Recording** - Capture tap, swipe, long_press to JSON with timestamps
- [x] **Action Playback** - Load recorded JSON and replay with timing
- [x] **Screenshot Diff** - Pixel-level PNG comparison with configurable tolerance
- [x] **GLFW Input Hooks** - Automatic mouse/key recording via forked backend

### Test Recording Format

```json
{
  "name": "Navigate to contacts and back",
  "recorded_at": "2024-01-16T12:00:00Z",
  "resolution": {"width": 677, "height": 1202},
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "wait", "duration": 500, "timestamp": 1200}
  ],
  "assertions": [
    {"after_action": 0, "type": "screen_is", "expected": "contacts"},
    {"after_action": 2, "type": "screen_is", "expected": "home"}
  ]
}
```

---

## Milestone 3: Virtual Hardware ✅ COMPLETE

**Goal**: Hardware-like APIs backed by game engine or real devices.

Implemented as part of the Lua Sandbox system. See [SANDBOX_MILESTONES.md](SANDBOX_MILESTONES.md) for details.

### Completed Components

| Component | Sandbox Milestone | Location |
|-----------|------------------|----------|
| Camera | Milestone 11 | `sandbox/camera_interface.cpp` |
| Microphone | Milestone 12 | `sandbox/microphone_interface.cpp` |
| Audio Output | Milestone 13 | `sandbox/audio_output.cpp` |
| Location/GPS | Milestone 14 | `sandbox/location_interface.cpp` |
| Sensors | Milestone 15 | `sandbox/sensor_interface.cpp` |
| Bluetooth | Milestone 16 | `sandbox/bluetooth_interface.cpp` |
| Filesystem | Milestone 7 | `sandbox/virtual_fs.cpp` |
| Network/HTTP | Milestone 9 | `sandbox/network_manager.cpp` |
| WebSocket | Milestone 10 | `sandbox/websocket_manager.cpp` |

---

## Milestone 4: App Sandboxing ✅ COMPLETE

**Goal**: Secure app runtime with defined package format.

Implemented with 20 sub-milestones covering security, APIs, and hardware interfaces. 149 security tests passing.

See:
- [SANDBOX_MILESTONES.md](SANDBOX_MILESTONES.md) - Implementation milestones
- [LUA-SANDBOX.md](LUA-SANDBOX.md) - Security features, permissions, APIs
- [APP-MANAGEMENT.md](APP-MANAGEMENT.md) - Package format, manifest, lifecycle

### Key Features

- **Lua Runtime**: Isolated `lua_State` per app with dangerous globals removed
- **Resource Limits**: Memory (16MB), CPU (instruction hooks), timers (100 max)
- **Permission System**: Normal/Dangerous/Signature categories with user gesture requirements
- **Virtual Filesystem**: Per-app storage with 50MB quota
- **SQLite Database**: Per-app with authorizer blocking dangerous operations
- **Network Security**: HTTPS required, private IP blocking, domain whitelists
- **Hardware Interfaces**: Camera, mic, location, sensors with mandatory indicators

---

## Milestone 5: WebRTC Bridge ❌ NOT STARTED

**Goal**: Cross-device communication via WebRTC.

### 5.1 Dependencies

- libdatachannel (add to vcpkg)

### 5.2 Communication Protocol

```json
{
  "type": "call" | "message" | "file" | "screen_share",
  "from": "phone_id",
  "to": "phone_id",
  "payload": { ... }
}
```

### 5.3 Use Cases

- Voice/video calls between virtual phones
- Text messaging across games
- File sharing
- Screen sharing

### 5.4 Components

- **WebRTCBridge** class for data channels
- **Signaling Server** (WebSocket-based)
- **Real Smartphone Bridge** (companion app)

---

## Milestone 6: System Apps 🔶 75% COMPLETE

### Completed Apps

| App | Features | Status |
|-----|----------|--------|
| Home | App grid, dock, navigation | ✅ Complete |
| Dialer | Keypad, call UI (mock) | ✅ Complete |
| Messages | Conversation list, chat view | ✅ Complete |
| Contacts | Contact list, search, detail | ✅ Complete |
| Settings | Display, sound, about | ✅ Complete |
| Browser | URL bar, placeholder | ✅ Complete |

### Remaining Apps

| App | Features | Status |
|-----|----------|--------|
| Store | Browse apps, install packages | ❌ UI only |
| Camera | Viewfinder, capture, gallery | ❌ UI only |
| Music/Audio | Playback, playlists | ❌ Outline only |

### App Data Persistence (Future)

- Contacts: JSON or SQLite storage
- Messages: Conversation history
- Settings: Preferences file
- Photos: Virtual filesystem

---

## Milestone 7: Game Integration ❌ NOT STARTED

**Goal**: Production-ready game engine plugins.

### 7.1 Unity Package

- [ ] Improved raycast interaction
- [ ] Hardware button simulation (volume, power)
- [ ] Virtual camera provider (RenderTexture → phone)
- [ ] Virtual microphone provider
- [ ] Virtual speaker provider

### 7.2 Unreal Plugin

- [ ] Full Binder client implementation
- [ ] Blueprint integration
- [ ] Same features as Unity

### 7.3 Documentation

- [ ] How to embed phone in game scene
- [ ] How to provide virtual hardware
- [ ] How to handle phone events

---

## Extended Features (Future)

| Feature | Description |
|---------|-------------|
| Notifications | System notification center, per-app channels |
| Widgets | Home screen widgets, live data |
| Gestures | Swipe up for home, swipe down for notifications |
| Themes | Dark/light mode, custom colors |
| Accessibility | Font size, high contrast, TTS |
| Multi-Window | Split-screen, picture-in-picture |
| Clipboard | Copy/paste between apps and game |
| Voice Assistant | Wake word, voice commands |
| Virtual Location | GPS from game, map integration |

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        VR Game (Unity/Unreal)                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────────┐  │
│  │ Phone Render │  │ Touch/Button │  │ Virtual Hardware Provider │  │
│  │   Viewport   │  │    Input     │  │ (Camera, Mic, Speakers)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────────┬─────────────┘  │
└─────────┼─────────────────┼────────────────────────┼────────────────┘
          │ Binder IPC      │ Binder IPC             │ Binder IPC
          ▼                 ▼                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Mosis Service                               │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │                         Kernel                               │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   RmlUi     │  │ App Runtime │  │   Virtual Hardware  │  │    │
│  │  │  Renderer   │  │   (Lua)     │  │      Abstraction    │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   App       │  │   IPC       │  │   libdatachannel    │  │    │
│  │  │  Manager    │  │  Router     │  │   (WebRTC Bridge)   │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
          │
          │ WebRTC (libdatachannel)
          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Real Smartphone / Other Devices                   │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Folder Structure

```
MosisService/
├── src/main/
│   ├── kernel/                    # Shared cross-platform kernel
│   │   ├── include/               # Platform abstractions
│   │   └── src/                   # Shared implementation
│   ├── cpp/                       # Android-specific native code
│   ├── java/                      # Kotlin/Java code
│   └── assets/                    # Shared UI assets
│       ├── apps/                  # System apps
│       │   ├── home/
│       │   ├── dialer/
│       │   ├── messages/
│       │   ├── contacts/
│       │   ├── settings/
│       │   ├── browser/
│       │   ├── store/             # TODO
│       │   └── camera/            # TODO
│       ├── ui/                    # Shared stylesheets
│       ├── scripts/               # Lua scripts
│       ├── icons/                 # TGA icons
│       └── fonts/                 # TTF fonts
│
├── designer/                      # Desktop designer
│   ├── src/
│   │   ├── testing/               # UI inspection, capture
│   │   └── ...
│   └── build/
│
├── designer-test/                 # Automated UI tests
│   ├── src/
│   └── build/
│
└── docs/                          # Documentation (future)
```

---

## Build Commands

```bash
# Android
./gradlew assembleDebug
./gradlew installDebug

# Desktop Designer
cd designer
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml

# Designer Tests
cd designer-test
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/designer-test.exe
```

---

## Current Sprint: Next Steps

### Priority 1: System Apps Completion
- [ ] Store app (UI only - browse, install)
- [ ] Camera app (UI + game engine texture)
- [ ] Music app (UI outline only)

### Priority 2: Game Engine Integration
- [x] MosisUnreal - Vulkan texture import working
- [ ] MosisUnreal - Blueprint API polish
- [ ] MosisVR (Unity) - Vulkan backend

### Priority 3: WebRTC Bridge (Milestone 5)
- [ ] libdatachannel integration
- [ ] Phone-to-phone communication
- [ ] Real smartphone bridge app

---

## Resources

### Material Design

- **Icons**: `D:\Dev\Mosis\MosisDesigner\material-design-icons`
  - 20 categories, SVG/PNG/Font formats
  - Browse at https://fonts.google.com/icons

- **Components**: `D:\Dev\Mosis\MosisDesigner\material-design-lite`
  - Reference implementation for design patterns

### Documentation

All documentation is in `docs/`:
- `CLAUDE.md` - Development guide
- `SANDBOX_MILESTONES.md` - Sandbox implementation details
- `ROADMAP.md` - This file

---

*Last updated: 2026-01-19*