# Mosis Virtual Smartphone Platform - Roadmap

## Vision

Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality.

### Key Use Cases

1. **In-Game Phone**: Players use a virtual phone in VR games for communication, utilities, entertainment
2. **Cross-Game Communication**: Phone identity persists across different games (unified contacts, messages)
3. **Real-World Bridge**: Connect to real smartphones via WebRTC for mixed-reality communication
4. **Virtual Hardware**: Game engine provides camera textures, audio, and other "hardware" to the phone OS
5. **Store Ecosystem**: Downloadable apps from a store, self-contained packages with isolation

### Integration Points

| Platform | Location | Status |
|----------|----------|--------|
| Unity Plugin | `D:\Dev\Mosis\Mosis Unity` | Binder client, raycast, viewport |
| Unreal Plugin | `D:\Dev\Mosis\Mosis Unreal` | WIP |
| Desktop Designer | `MosisService/designer/` | Complete |
| Designer Tests | `MosisService/designer-test/` | Mostly complete |

---

## Milestones Overview

| # | Milestone | Status | Description |
|---|-----------|--------|-------------|
| 1 | Cross-Platform Kernel | ✅ Complete | Desktop designer with shared kernel code |
| 2 | Testing Framework | ✅ Complete | Automated UI testing and inspection |
| 3 | Virtual Hardware | ❌ Not started | Camera, mic, speaker, filesystem APIs |
| 4 | App Sandboxing | ❌ Not started | Lua/WASM runtime, package format |
| 5 | WebRTC Bridge | ❌ Not started | Phone-to-phone communication |
| 6 | System Apps | 🔶 75% | Core phone apps |
| 7 | Game Integration | ❌ Not started | Unity/Unreal plugin polish |

---

## Milestone 1: Cross-Platform Kernel ✅ COMPLETE

**Goal**: Desktop designer with shared kernel code for rapid UI iteration.

### Completed Tasks

- [x] Platform abstraction layer (`src/main/kernel/include/`)
  - `platform.h` - IPlatform, IGraphicsContext, IRenderTarget
  - `service_interface.h` - IKernel, IServiceListener
  - `file_interface.h` - IFileInterface extending Rml::FileInterface
- [x] Desktop designer project (`designer/`)
  - GLFW + OpenGL 3.3 context
  - Hot-reload on RML/RCSS/Lua changes
  - Mouse input maps to touch events
- [x] UI designs imported from MosisDesigner
  - Theme and component stylesheets
  - Navigation system (Lua)
  - Fonts and icons
- [x] Android build unchanged and working

---

## Milestone 2: Testing Automation ✅ COMPLETE

**Goal**: Automated UI testing for rapid iteration and AI agent verification.

### Completed Tasks

- [x] UI hierarchy dumping to JSON
- [x] PNG screenshot capture (F12 key)
- [x] Agent-compatible JSON test results
- [x] `designer-test` executable
  - WindowController (Windows SendInput API)
  - HierarchyReader (element lookup by ID/class)
  - LogParser (navigation event verification)
- [x] All 5 navigation tests passing
- [x] Android event injection via ADB broadcast
- [x] **Action Recording** - Capture tap, swipe, long_press to JSON with timestamps
- [x] **Action Playback** - Load recorded JSON and replay with timing
- [x] **Screenshot Diff** - Pixel-level PNG comparison with configurable tolerance
- [x] **GLFW Input Hooks** - Automatic mouse/key recording via forked backend

### Test Recording Format

```json
{
  "name": "Navigate to contacts and back",
  "recorded_at": "2024-01-16T12:00:00Z",
  "resolution": {"width": 677, "height": 1202},
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "wait", "duration": 500, "timestamp": 1200}
  ],
  "assertions": [
    {"after_action": 0, "type": "screen_is", "expected": "contacts"},
    {"after_action": 2, "type": "screen_is", "expected": "home"}
  ]
}
```

---

## Milestone 3: Virtual Hardware ❌ NOT STARTED

**Goal**: Hardware-like APIs backed by game engine or real devices.

### 3.1 Camera Interface

```cpp
class ICamera {
    virtual void RequestFrame(FrameCallback callback) = 0;
    virtual void SetResolution(int width, int height) = 0;
    virtual bool IsAvailable() const = 0;
};
```

**Implementations**:
- **Game Mode**: Receives texture from Unity/Unreal
- **Desktop Mode**: System webcam (optional)
- **Android Test Mode**: SharedTexture from MainActivity
- **Mock Mode**: Test patterns

### 3.2 Microphone Interface

```cpp
class IMicrophone {
    virtual void StartCapture(AudioCallback callback) = 0;
    virtual void StopCapture() = 0;
    virtual bool IsAvailable() const = 0;
};
```

### 3.3 Speaker Interface

```cpp
class ISpeaker {
    virtual void PlayAudio(AudioBuffer buffer) = 0;
    virtual void SetVolume(float volume) = 0;
};
```

### 3.4 Filesystem Interface

```cpp
class IFileSystem {
    virtual FileHandle Open(const std::string& path, Mode mode) = 0;
    virtual std::vector<std::string> List(const std::string& dir) = 0;
    virtual bool CreateDirectory(const std::string& path) = 0;
    virtual bool Delete(const std::string& path) = 0;
};
```

### 3.5 Network Interface

```cpp
class INetwork {
    virtual HttpResponse Fetch(const HttpRequest& request) = 0;
    virtual WebSocket Connect(const std::string& url) = 0;
    virtual bool IsOnline() const = 0;
};
```

---

## Milestone 4: App Sandboxing ❌ NOT STARTED

**Goal**: Secure app runtime with defined package format.

### 4.1 Runtime Decision

| Aspect | Lua | WASM |
|--------|-----|------|
| Isolation | Weak | Strong |
| Performance | Good for UI | Near-native |
| RmlUi Integration | Native | Needs bridge |
| Ecosystem | Small | Large |

**Recommendation**: Hybrid approach
- Lua for UI scripting (RmlUi integration)
- WASM for app logic needing isolation (future)

### 4.2 Package Format (.mpkg)

```
myapp.mpkg/
├── manifest.json        # Metadata, permissions, entry
├── ui/
│   ├── main.rml
│   ├── styles.rcss
│   └── scripts/
├── assets/
│   ├── icons/
│   └── images/
└── data/
```

### 4.3 Manifest Schema

```json
{
  "id": "com.example.myapp",
  "name": "My App",
  "version": "1.0.0",
  "permissions": ["camera", "network", "storage"],
  "entry": "ui/main.rml",
  "icon": "assets/icons/app.png"
}
```

### 4.4 App Lifecycle

```
INSTALLED → LAUNCHING → RUNNING → PAUSED → STOPPED → UNINSTALLED
```

---

## Milestone 5: WebRTC Bridge ❌ NOT STARTED

**Goal**: Cross-device communication via WebRTC.

### 5.1 Dependencies

- libdatachannel (add to vcpkg)

### 5.2 Communication Protocol

```json
{
  "type": "call" | "message" | "file" | "screen_share",
  "from": "phone_id",
  "to": "phone_id",
  "payload": { ... }
}
```

### 5.3 Use Cases

- Voice/video calls between virtual phones
- Text messaging across games
- File sharing
- Screen sharing

### 5.4 Components

- **WebRTCBridge** class for data channels
- **Signaling Server** (WebSocket-based)
- **Real Smartphone Bridge** (companion app)

---

## Milestone 6: System Apps 🔶 75% COMPLETE

### Completed Apps

| App | Features | Status |
|-----|----------|--------|
| Home | App grid, dock, navigation | ✅ Complete |
| Dialer | Keypad, call UI (mock) | ✅ Complete |
| Messages | Conversation list, chat view | ✅ Complete |
| Contacts | Contact list, search, detail | ✅ Complete |
| Settings | Display, sound, about | ✅ Complete |
| Browser | URL bar, placeholder | ✅ Complete |

### Remaining Apps

| App | Features | Status |
|-----|----------|--------|
| Store | Browse apps, install packages | ❌ UI only |
| Camera | Viewfinder, capture, gallery | ❌ UI only |
| Music/Audio | Playback, playlists | ❌ Outline only |

### App Data Persistence (Future)

- Contacts: JSON or SQLite storage
- Messages: Conversation history
- Settings: Preferences file
- Photos: Virtual filesystem

---

## Milestone 7: Game Integration ❌ NOT STARTED

**Goal**: Production-ready game engine plugins.

### 7.1 Unity Package

- [ ] Improved raycast interaction
- [ ] Hardware button simulation (volume, power)
- [ ] Virtual camera provider (RenderTexture → phone)
- [ ] Virtual microphone provider
- [ ] Virtual speaker provider

### 7.2 Unreal Plugin

- [ ] Full Binder client implementation
- [ ] Blueprint integration
- [ ] Same features as Unity

### 7.3 Documentation

- [ ] How to embed phone in game scene
- [ ] How to provide virtual hardware
- [ ] How to handle phone events

---

## Extended Features (Future)

| Feature | Description |
|---------|-------------|
| Notifications | System notification center, per-app channels |
| Widgets | Home screen widgets, live data |
| Gestures | Swipe up for home, swipe down for notifications |
| Themes | Dark/light mode, custom colors |
| Accessibility | Font size, high contrast, TTS |
| Multi-Window | Split-screen, picture-in-picture |
| Clipboard | Copy/paste between apps and game |
| Voice Assistant | Wake word, voice commands |
| Virtual Location | GPS from game, map integration |

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        VR Game (Unity/Unreal)                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────────┐  │
│  │ Phone Render │  │ Touch/Button │  │ Virtual Hardware Provider │  │
│  │   Viewport   │  │    Input     │  │ (Camera, Mic, Speakers)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────────┬─────────────┘  │
└─────────┼─────────────────┼────────────────────────┼────────────────┘
          │ Binder IPC      │ Binder IPC             │ Binder IPC
          ▼                 ▼                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Mosis Service                               │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │                         Kernel                               │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   RmlUi     │  │ App Runtime │  │   Virtual Hardware  │  │    │
│  │  │  Renderer   │  │   (Lua)     │  │      Abstraction    │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   App       │  │   IPC       │  │   libdatachannel    │  │    │
│  │  │  Manager    │  │  Router     │  │   (WebRTC Bridge)   │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
          │
          │ WebRTC (libdatachannel)
          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Real Smartphone / Other Devices                   │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Folder Structure

```
MosisService/
├── src/main/
│   ├── kernel/                    # Shared cross-platform kernel
│   │   ├── include/               # Platform abstractions
│   │   └── src/                   # Shared implementation
│   ├── cpp/                       # Android-specific native code
│   ├── java/                      # Kotlin/Java code
│   └── assets/                    # Shared UI assets
│       ├── apps/                  # System apps
│       │   ├── home/
│       │   ├── dialer/
│       │   ├── messages/
│       │   ├── contacts/
│       │   ├── settings/
│       │   ├── browser/
│       │   ├── store/             # TODO
│       │   └── camera/            # TODO
│       ├── ui/                    # Shared stylesheets
│       ├── scripts/               # Lua scripts
│       ├── icons/                 # TGA icons
│       └── fonts/                 # TTF fonts
│
├── designer/                      # Desktop designer
│   ├── src/
│   │   ├── testing/               # UI inspection, capture
│   │   └── ...
│   └── build/
│
├── designer-test/                 # Automated UI tests
│   ├── src/
│   └── build/
│
└── docs/                          # Documentation (future)
```

---

## Build Commands

```bash
# Android
./gradlew assembleDebug
./gradlew installDebug

# Desktop Designer
cd designer
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml

# Designer Tests
cd designer-test
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/designer-test.exe
```

---

## Current Sprint: Complete Partial Tasks

### Priority 1: Testing Framework Completion ✅ DONE
- [x] Action recording (capture interactions to JSON)
- [x] Action playback (replay with timing)
- [x] Screenshot diff (visual regression)
- [x] GLFW input hooks for automatic recording

### Priority 2: Remaining System Apps
- [ ] Store app (UI only - browse, install)
- [ ] Camera app (UI + shared texture from MainActivity)
- [ ] Music app (UI outline only)

### Priority 3: App Data Persistence
- [ ] JSON/SQLite storage layer
- [ ] Contact CRUD operations
- [ ] Message history
- [ ] Settings persistence

---

## Resources

### Material Design

- **Icons**: `D:\Dev\Mosis\MosisDesigner\material-design-icons`
  - 20 categories, SVG/PNG/Font formats
  - Browse at https://fonts.google.com/icons

- **Components**: `D:\Dev\Mosis\MosisDesigner\material-design-lite`
  - Reference implementation for design patterns

### Documentation

- `CLAUDE.md` - Development guide
- `TESTING.md` - Testing framework documentation
- `ROADMAP.md` - This file

---

*Last updated: 2026-01-16*