Files

omigamedev bbf1638f20 update roadmap with completed milestones, trim SANDBOX.md to reference milestone files

2026-01-19 09:07:31 +01:00

15 KiB

Raw Permalink Blame History

Mosis Virtual Smartphone Platform - Roadmap

Vision

Mosis is a virtual smartphone OS for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality.

Key Use Cases

In-Game Phone: Players use a virtual phone in VR games for communication, utilities, entertainment
Cross-Game Communication: Phone identity persists across different games (unified contacts, messages)
Real-World Bridge: Connect to real smartphones via WebRTC for mixed-reality communication
Virtual Hardware: Game engine provides camera textures, audio, and other "hardware" to the phone OS
Store Ecosystem: Downloadable apps from a store, self-contained packages with isolation

Integration Points

Platform	Location	Status
Unity Plugin	`D:\Dev\Mosis\Mosis Unity`	Binder client, raycast, viewport
Unreal Plugin	`D:\Dev\Mosis\Mosis Unreal`	WIP
Desktop Designer	`MosisService/designer/`	Complete
Designer Tests	`MosisService/designer-test/`	Mostly complete

Milestones Overview

#	Milestone	Status	Description
1	Cross-Platform Kernel	✅ Complete	Desktop designer with shared kernel code
2	Testing Framework	✅ Complete	Automated UI testing and inspection
3	Virtual Hardware	✅ Complete	Camera, mic, speaker, filesystem APIs (via sandbox)
4	App Sandboxing	✅ Complete	Lua runtime, package format, 149 security tests
5	WebRTC Bridge	❌ Not started	Phone-to-phone communication
6	System Apps	🔶 75%	Core phone apps
7	Game Integration	🔶 In Progress	Unity/Unreal plugins (Vulkan working)

Milestone 1: Cross-Platform Kernel ✅ COMPLETE

Goal: Desktop designer with shared kernel code for rapid UI iteration.

Completed Tasks

Platform abstraction layer (src/main/kernel/include/)
- platform.h - IPlatform, IGraphicsContext, IRenderTarget
- service_interface.h - IKernel, IServiceListener
- file_interface.h - IFileInterface extending Rml::FileInterface
Desktop designer project (designer/)
- GLFW + OpenGL 3.3 context
- Hot-reload on RML/RCSS/Lua changes
- Mouse input maps to touch events
UI designs imported from MosisDesigner
- Theme and component stylesheets
- Navigation system (Lua)
- Fonts and icons
Android build unchanged and working

Milestone 2: Testing Automation ✅ COMPLETE

Goal: Automated UI testing for rapid iteration and AI agent verification.

Completed Tasks

UI hierarchy dumping to JSON
PNG screenshot capture (F12 key)
Agent-compatible JSON test results
designer-test executable
- WindowController (Windows SendInput API)
- HierarchyReader (element lookup by ID/class)
- LogParser (navigation event verification)
All 5 navigation tests passing
Android event injection via ADB broadcast
Action Recording - Capture tap, swipe, long_press to JSON with timestamps
Action Playback - Load recorded JSON and replay with timing
Screenshot Diff - Pixel-level PNG comparison with configurable tolerance
GLFW Input Hooks - Automatic mouse/key recording via forked backend

Test Recording Format

{
  "name": "Navigate to contacts and back",
  "recorded_at": "2024-01-16T12:00:00Z",
  "resolution": {"width": 677, "height": 1202},
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "wait", "duration": 500, "timestamp": 1200}
  ],
  "assertions": [
    {"after_action": 0, "type": "screen_is", "expected": "contacts"},
    {"after_action": 2, "type": "screen_is", "expected": "home"}
  ]
}

Milestone 3: Virtual Hardware ✅ COMPLETE

Goal: Hardware-like APIs backed by game engine or real devices.

Implemented as part of the Lua Sandbox system. See SANDBOX_MILESTONES.md for details.

Completed Components

Component	Sandbox Milestone	Location
Camera	Milestone 11	`sandbox/camera_interface.cpp`
Microphone	Milestone 12	`sandbox/microphone_interface.cpp`
Audio Output	Milestone 13	`sandbox/audio_output.cpp`
Location/GPS	Milestone 14	`sandbox/location_interface.cpp`
Sensors	Milestone 15	`sandbox/sensor_interface.cpp`
Bluetooth	Milestone 16	`sandbox/bluetooth_interface.cpp`
Filesystem	Milestone 7	`sandbox/virtual_fs.cpp`
Network/HTTP	Milestone 9	`sandbox/network_manager.cpp`
WebSocket	Milestone 10	`sandbox/websocket_manager.cpp`

Milestone 4: App Sandboxing ✅ COMPLETE

Goal: Secure app runtime with defined package format.

Implemented with 20 sub-milestones covering security, APIs, and hardware interfaces. 149 security tests passing.

See:

SANDBOX_MILESTONES.md - Implementation milestones
LUA-SANDBOX.md - Security features, permissions, APIs
APP-MANAGEMENT.md - Package format, manifest, lifecycle

Key Features

Lua Runtime: Isolated lua_State per app with dangerous globals removed
Resource Limits: Memory (16MB), CPU (instruction hooks), timers (100 max)
Permission System: Normal/Dangerous/Signature categories with user gesture requirements
Virtual Filesystem: Per-app storage with 50MB quota
SQLite Database: Per-app with authorizer blocking dangerous operations
Network Security: HTTPS required, private IP blocking, domain whitelists
Hardware Interfaces: Camera, mic, location, sensors with mandatory indicators

Milestone 5: WebRTC Bridge ❌ NOT STARTED

Goal: Cross-device communication via WebRTC.

5.1 Dependencies

libdatachannel (add to vcpkg)

5.2 Communication Protocol

{
  "type": "call" | "message" | "file" | "screen_share",
  "from": "phone_id",
  "to": "phone_id",
  "payload": { ... }
}

5.3 Use Cases

Voice/video calls between virtual phones
Text messaging across games
File sharing
Screen sharing

5.4 Components

WebRTCBridge class for data channels
Signaling Server (WebSocket-based)
Real Smartphone Bridge (companion app)

Milestone 6: System Apps 🔶 75% COMPLETE

Completed Apps

App	Features	Status
Home	App grid, dock, navigation	✅ Complete
Dialer	Keypad, call UI (mock)	✅ Complete
Messages	Conversation list, chat view	✅ Complete
Contacts	Contact list, search, detail	✅ Complete
Settings	Display, sound, about	✅ Complete
Browser	URL bar, placeholder	✅ Complete

Remaining Apps

App	Features	Status
Store	Browse apps, install packages	❌ UI only
Camera	Viewfinder, capture, gallery	❌ UI only
Music/Audio	Playback, playlists	❌ Outline only

App Data Persistence (Future)

Contacts: JSON or SQLite storage
Messages: Conversation history
Settings: Preferences file
Photos: Virtual filesystem

Milestone 7: Game Integration ❌ NOT STARTED

Goal: Production-ready game engine plugins.

7.1 Unity Package

Improved raycast interaction
Hardware button simulation (volume, power)
Virtual camera provider (RenderTexture → phone)
Virtual microphone provider
Virtual speaker provider

7.2 Unreal Plugin

Full Binder client implementation
Blueprint integration
Same features as Unity

7.3 Documentation

How to embed phone in game scene
How to provide virtual hardware
How to handle phone events

Extended Features (Future)

Feature	Description
Notifications	System notification center, per-app channels
Widgets	Home screen widgets, live data
Gestures	Swipe up for home, swipe down for notifications
Themes	Dark/light mode, custom colors
Accessibility	Font size, high contrast, TTS
Multi-Window	Split-screen, picture-in-picture
Clipboard	Copy/paste between apps and game
Voice Assistant	Wake word, voice commands
Virtual Location	GPS from game, map integration

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        VR Game (Unity/Unreal)                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────────┐  │
│  │ Phone Render │  │ Touch/Button │  │ Virtual Hardware Provider │  │
│  │   Viewport   │  │    Input     │  │ (Camera, Mic, Speakers)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────────┬─────────────┘  │
└─────────┼─────────────────┼────────────────────────┼────────────────┘
          │ Binder IPC      │ Binder IPC             │ Binder IPC
          ▼                 ▼                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Mosis Service                               │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │                         Kernel                               │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   RmlUi     │  │ App Runtime │  │   Virtual Hardware  │  │    │
│  │  │  Renderer   │  │   (Lua)     │  │      Abstraction    │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   App       │  │   IPC       │  │   libdatachannel    │  │    │
│  │  │  Manager    │  │  Router     │  │   (WebRTC Bridge)   │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
          │
          │ WebRTC (libdatachannel)
          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Real Smartphone / Other Devices                   │
└─────────────────────────────────────────────────────────────────────┘

Folder Structure

MosisService/
├── src/main/
│   ├── kernel/                    # Shared cross-platform kernel
│   │   ├── include/               # Platform abstractions
│   │   └── src/                   # Shared implementation
│   ├── cpp/                       # Android-specific native code
│   ├── java/                      # Kotlin/Java code
│   └── assets/                    # Shared UI assets
│       ├── apps/                  # System apps
│       │   ├── home/
│       │   ├── dialer/
│       │   ├── messages/
│       │   ├── contacts/
│       │   ├── settings/
│       │   ├── browser/
│       │   ├── store/             # TODO
│       │   └── camera/            # TODO
│       ├── ui/                    # Shared stylesheets
│       ├── scripts/               # Lua scripts
│       ├── icons/                 # TGA icons
│       └── fonts/                 # TTF fonts
│
├── designer/                      # Desktop designer
│   ├── src/
│   │   ├── testing/               # UI inspection, capture
│   │   └── ...
│   └── build/
│
├── designer-test/                 # Automated UI tests
│   ├── src/
│   └── build/
│
└── docs/                          # Documentation (future)

Build Commands

# Android
./gradlew assembleDebug
./gradlew installDebug

# Desktop Designer
cd designer
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml

# Designer Tests
cd designer-test
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/designer-test.exe

Current Sprint: Next Steps

Priority 1: System Apps Completion

Store app (UI only - browse, install)
Camera app (UI + game engine texture)
Music app (UI outline only)

Priority 2: Game Engine Integration

MosisUnreal - Vulkan texture import working
MosisUnreal - Blueprint API polish
MosisVR (Unity) - Vulkan backend

Priority 3: WebRTC Bridge (Milestone 5)

libdatachannel integration
Phone-to-phone communication
Real smartphone bridge app

Resources

Material Design

Icons: D:\Dev\Mosis\MosisDesigner\material-design-icons
- 20 categories, SVG/PNG/Font formats
- Browse at https://fonts.google.com/icons
Components: D:\Dev\Mosis\MosisDesigner\material-design-lite
- Reference implementation for design patterns

Documentation

All documentation is in docs/:

CLAUDE.md - Development guide
SANDBOX_MILESTONES.md - Sandbox implementation details
ROADMAP.md - This file

Last updated: 2026-01-19

15 KiB Raw Permalink Blame History

Mosis Virtual Smartphone Platform - Roadmap

Vision

Key Use Cases

Integration Points

Milestones Overview

Milestone 1: Cross-Platform Kernel ✅ COMPLETE

Completed Tasks

Milestone 2: Testing Automation ✅ COMPLETE

Completed Tasks

Test Recording Format

Milestone 3: Virtual Hardware ✅ COMPLETE

Completed Components

Milestone 4: App Sandboxing ✅ COMPLETE

Key Features

Milestone 5: WebRTC Bridge ❌ NOT STARTED

5.1 Dependencies

5.2 Communication Protocol

5.3 Use Cases

5.4 Components

Milestone 6: System Apps 🔶 75% COMPLETE

Completed Apps

Remaining Apps

App Data Persistence (Future)

Milestone 7: Game Integration ❌ NOT STARTED

7.1 Unity Package

7.2 Unreal Plugin

7.3 Documentation

Extended Features (Future)

Architecture

Folder Structure

Build Commands

Current Sprint: Next Steps

Priority 1: System Apps Completion

Priority 2: Game Engine Integration

Priority 3: WebRTC Bridge (Milestone 5)

Resources

Material Design

Documentation

15 KiB

Raw Permalink Blame History