Files

2026-01-16 13:53:59 +01:00

16 KiB

Raw Blame History

Mosis Virtual Smartphone Platform - Roadmap

Vision

Mosis is a virtual smartphone OS for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality.

Key Use Cases

In-Game Phone: Players use a virtual phone in VR games for communication, utilities, entertainment
Cross-Game Communication: Phone identity persists across different games (unified contacts, messages)
Real-World Bridge: Connect to real smartphones via WebRTC for mixed-reality communication
Virtual Hardware: Game engine provides camera textures, audio, and other "hardware" to the phone OS
Store Ecosystem: Downloadable apps from a store, self-contained packages with isolation

Integration Points

Platform	Location	Status
Unity Plugin	`D:\Dev\Mosis\Mosis Unity`	Binder client, raycast, viewport
Unreal Plugin	`D:\Dev\Mosis\Mosis Unreal`	WIP
Desktop Designer	`MosisService/designer/`	Complete
Designer Tests	`MosisService/designer-test/`	Mostly complete

Milestones Overview

#	Milestone	Status	Description
1	Cross-Platform Kernel	✅ Complete	Desktop designer with shared kernel code
2	Testing Framework	🔶 80%	Automated UI testing and inspection
3	Virtual Hardware	❌ Not started	Camera, mic, speaker, filesystem APIs
4	App Sandboxing	❌ Not started	Lua/WASM runtime, package format
5	WebRTC Bridge	❌ Not started	Phone-to-phone communication
6	System Apps	🔶 75%	Core phone apps
7	Game Integration	❌ Not started	Unity/Unreal plugin polish

Milestone 1: Cross-Platform Kernel ✅ COMPLETE

Goal: Desktop designer with shared kernel code for rapid UI iteration.

Completed Tasks

Platform abstraction layer (src/main/kernel/include/)
- platform.h - IPlatform, IGraphicsContext, IRenderTarget
- service_interface.h - IKernel, IServiceListener
- file_interface.h - IFileInterface extending Rml::FileInterface
Desktop designer project (designer/)
- GLFW + OpenGL 3.3 context
- Hot-reload on RML/RCSS/Lua changes
- Mouse input maps to touch events
UI designs imported from MosisDesigner
- Theme and component stylesheets
- Navigation system (Lua)
- Fonts and icons
Android build unchanged and working

Milestone 2: Testing Automation 🔶 80% COMPLETE

Goal: Automated UI testing for rapid iteration and AI agent verification.

Completed Tasks

UI hierarchy dumping to JSON
PNG screenshot capture (F12 key)
Agent-compatible JSON test results
designer-test executable
- WindowController (Windows SendInput API)
- HierarchyReader (element lookup by ID/class)
- LogParser (navigation event verification)
All 5 navigation tests passing
Android event injection via ADB broadcast

Remaining Tasks

Action Recording
- Capture tap, swipe, long_press to JSON
- Record timestamps for replay timing
- Save to test-recordings/*.json
Action Playback
- Load recorded JSON
- Replay with timing
- Capture results at each step
Screenshot Diff
- Compare two PNG screenshots
- Highlight pixel differences
- Report diff percentage

Test Recording Format

{
  "name": "Navigate to contacts and back",
  "recorded_at": "2024-01-16T12:00:00Z",
  "resolution": {"width": 677, "height": 1202},
  "actions": [
    {"type": "tap", "x": 413, "y": 1174, "timestamp": 0},
    {"type": "wait", "duration": 1000, "timestamp": 100},
    {"type": "tap", "x": 40, "y": 28, "timestamp": 1100},
    {"type": "wait", "duration": 500, "timestamp": 1200}
  ],
  "assertions": [
    {"after_action": 0, "type": "screen_is", "expected": "contacts"},
    {"after_action": 2, "type": "screen_is", "expected": "home"}
  ]
}

Milestone 3: Virtual Hardware ❌ NOT STARTED

Goal: Hardware-like APIs backed by game engine or real devices.

3.1 Camera Interface

class ICamera {
    virtual void RequestFrame(FrameCallback callback) = 0;
    virtual void SetResolution(int width, int height) = 0;
    virtual bool IsAvailable() const = 0;
};

Implementations:

Game Mode: Receives texture from Unity/Unreal
Desktop Mode: System webcam (optional)
Android Test Mode: SharedTexture from MainActivity
Mock Mode: Test patterns

3.2 Microphone Interface

class IMicrophone {
    virtual void StartCapture(AudioCallback callback) = 0;
    virtual void StopCapture() = 0;
    virtual bool IsAvailable() const = 0;
};

3.3 Speaker Interface

class ISpeaker {
    virtual void PlayAudio(AudioBuffer buffer) = 0;
    virtual void SetVolume(float volume) = 0;
};

3.4 Filesystem Interface

class IFileSystem {
    virtual FileHandle Open(const std::string& path, Mode mode) = 0;
    virtual std::vector<std::string> List(const std::string& dir) = 0;
    virtual bool CreateDirectory(const std::string& path) = 0;
    virtual bool Delete(const std::string& path) = 0;
};

3.5 Network Interface

class INetwork {
    virtual HttpResponse Fetch(const HttpRequest& request) = 0;
    virtual WebSocket Connect(const std::string& url) = 0;
    virtual bool IsOnline() const = 0;
};

Milestone 4: App Sandboxing ❌ NOT STARTED

Goal: Secure app runtime with defined package format.

4.1 Runtime Decision

Aspect	Lua	WASM
Isolation	Weak	Strong
Performance	Good for UI	Near-native
RmlUi Integration	Native	Needs bridge
Ecosystem	Small	Large

Recommendation: Hybrid approach

Lua for UI scripting (RmlUi integration)
WASM for app logic needing isolation (future)

4.2 Package Format (.mpkg)

myapp.mpkg/
├── manifest.json        # Metadata, permissions, entry
├── ui/
│   ├── main.rml
│   ├── styles.rcss
│   └── scripts/
├── assets/
│   ├── icons/
│   └── images/
└── data/

4.3 Manifest Schema

{
  "id": "com.example.myapp",
  "name": "My App",
  "version": "1.0.0",
  "permissions": ["camera", "network", "storage"],
  "entry": "ui/main.rml",
  "icon": "assets/icons/app.png"
}

4.4 App Lifecycle

INSTALLED → LAUNCHING → RUNNING → PAUSED → STOPPED → UNINSTALLED

Milestone 5: WebRTC Bridge ❌ NOT STARTED

Goal: Cross-device communication via WebRTC.

5.1 Dependencies

libdatachannel (add to vcpkg)

5.2 Communication Protocol

{
  "type": "call" | "message" | "file" | "screen_share",
  "from": "phone_id",
  "to": "phone_id",
  "payload": { ... }
}

5.3 Use Cases

Voice/video calls between virtual phones
Text messaging across games
File sharing
Screen sharing

5.4 Components

WebRTCBridge class for data channels
Signaling Server (WebSocket-based)
Real Smartphone Bridge (companion app)

Milestone 6: System Apps 🔶 75% COMPLETE

Completed Apps

App	Features	Status
Home	App grid, dock, navigation	✅ Complete
Dialer	Keypad, call UI (mock)	✅ Complete
Messages	Conversation list, chat view	✅ Complete
Contacts	Contact list, search, detail	✅ Complete
Settings	Display, sound, about	✅ Complete
Browser	URL bar, placeholder	✅ Complete

Remaining Apps

App	Features	Status
Store	Browse apps, install packages	❌ UI only
Camera	Viewfinder, capture, gallery	❌ UI only
Music/Audio	Playback, playlists	❌ Outline only

App Data Persistence (Future)

Contacts: JSON or SQLite storage
Messages: Conversation history
Settings: Preferences file
Photos: Virtual filesystem

Milestone 7: Game Integration ❌ NOT STARTED

Goal: Production-ready game engine plugins.

7.1 Unity Package

Improved raycast interaction
Hardware button simulation (volume, power)
Virtual camera provider (RenderTexture → phone)
Virtual microphone provider
Virtual speaker provider

7.2 Unreal Plugin

Full Binder client implementation
Blueprint integration
Same features as Unity

7.3 Documentation

How to embed phone in game scene
How to provide virtual hardware
How to handle phone events

Extended Features (Future)

Feature	Description
Notifications	System notification center, per-app channels
Widgets	Home screen widgets, live data
Gestures	Swipe up for home, swipe down for notifications
Themes	Dark/light mode, custom colors
Accessibility	Font size, high contrast, TTS
Multi-Window	Split-screen, picture-in-picture
Clipboard	Copy/paste between apps and game
Voice Assistant	Wake word, voice commands
Virtual Location	GPS from game, map integration

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        VR Game (Unity/Unreal)                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────────┐  │
│  │ Phone Render │  │ Touch/Button │  │ Virtual Hardware Provider │  │
│  │   Viewport   │  │    Input     │  │ (Camera, Mic, Speakers)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────────┬─────────────┘  │
└─────────┼─────────────────┼────────────────────────┼────────────────┘
          │ Binder IPC      │ Binder IPC             │ Binder IPC
          ▼                 ▼                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Mosis Service                               │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │                         Kernel                               │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   RmlUi     │  │ App Runtime │  │   Virtual Hardware  │  │    │
│  │  │  Renderer   │  │   (Lua)     │  │      Abstraction    │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │    │
│  │  │   App       │  │   IPC       │  │   libdatachannel    │  │    │
│  │  │  Manager    │  │  Router     │  │   (WebRTC Bridge)   │  │    │
│  │  └─────────────┘  └─────────────┘  └─────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
          │
          │ WebRTC (libdatachannel)
          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Real Smartphone / Other Devices                   │
└─────────────────────────────────────────────────────────────────────┘

Folder Structure

MosisService/
├── src/main/
│   ├── kernel/                    # Shared cross-platform kernel
│   │   ├── include/               # Platform abstractions
│   │   └── src/                   # Shared implementation
│   ├── cpp/                       # Android-specific native code
│   ├── java/                      # Kotlin/Java code
│   └── assets/                    # Shared UI assets
│       ├── apps/                  # System apps
│       │   ├── home/
│       │   ├── dialer/
│       │   ├── messages/
│       │   ├── contacts/
│       │   ├── settings/
│       │   ├── browser/
│       │   ├── store/             # TODO
│       │   └── camera/            # TODO
│       ├── ui/                    # Shared stylesheets
│       ├── scripts/               # Lua scripts
│       ├── icons/                 # TGA icons
│       └── fonts/                 # TTF fonts
│
├── designer/                      # Desktop designer
│   ├── src/
│   │   ├── testing/               # UI inspection, capture
│   │   └── ...
│   └── build/
│
├── designer-test/                 # Automated UI tests
│   ├── src/
│   └── build/
│
└── docs/                          # Documentation (future)

Build Commands

# Android
./gradlew assembleDebug
./gradlew installDebug

# Desktop Designer
cd designer
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml

# Designer Tests
cd designer-test
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Debug
./build/Debug/designer-test.exe

Current Sprint: Complete Partial Tasks

Priority 1: Testing Framework Completion

Action recording (capture interactions to JSON)
Action playback (replay with timing)
Screenshot diff (visual regression)

Priority 2: Remaining System Apps

Store app (UI only - browse, install)
Camera app (UI + shared texture from MainActivity)
Music app (UI outline only)

Priority 3: App Data Persistence

JSON/SQLite storage layer
Contact CRUD operations
Message history
Settings persistence

Resources

Material Design

Icons: D:\Dev\Mosis\MosisDesigner\material-design-icons
- 20 categories, SVG/PNG/Font formats
- Browse at https://fonts.google.com/icons
Components: D:\Dev\Mosis\MosisDesigner\material-design-lite
- Reference implementation for design patterns

Documentation

CLAUDE.md - Development guide
TESTING.md - Testing framework documentation
ROADMAP.md - This file

Last updated: 2024-01-16

16 KiB Raw Blame History

Mosis Virtual Smartphone Platform - Roadmap

Vision

Key Use Cases

Integration Points

Milestones Overview

Milestone 1: Cross-Platform Kernel ✅ COMPLETE

Completed Tasks

Milestone 2: Testing Automation 🔶 80% COMPLETE

Completed Tasks

Remaining Tasks

Test Recording Format

Milestone 3: Virtual Hardware ❌ NOT STARTED

3.1 Camera Interface

3.2 Microphone Interface

3.3 Speaker Interface

3.4 Filesystem Interface

3.5 Network Interface

Milestone 4: App Sandboxing ❌ NOT STARTED

4.1 Runtime Decision

4.2 Package Format (.mpkg)

4.3 Manifest Schema

4.4 App Lifecycle

Milestone 5: WebRTC Bridge ❌ NOT STARTED

5.1 Dependencies

5.2 Communication Protocol

5.3 Use Cases

5.4 Components

Milestone 6: System Apps 🔶 75% COMPLETE

Completed Apps

Remaining Apps

App Data Persistence (Future)

Milestone 7: Game Integration ❌ NOT STARTED

7.1 Unity Package

7.2 Unreal Plugin

7.3 Documentation

Extended Features (Future)

Architecture

Folder Structure

Build Commands

Current Sprint: Complete Partial Tasks

Priority 1: Testing Framework Completion

Priority 2: Remaining System Apps

Priority 3: App Data Persistence

Resources

Material Design

Documentation

16 KiB

Raw Blame History