diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..369c138 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,475 @@ +# Mosis Virtual Smartphone Platform - Roadmap + +## Vision + +Mosis is a **virtual smartphone OS** for VR games and applications. It provides a phone-like device that users interact with inside VR environments, running virtual apps with real smartphone functionality. + +### Key Use Cases + +1. **In-Game Phone**: Players use a virtual phone in VR games for communication, utilities, entertainment +2. **Cross-Game Communication**: Phone identity persists across different games (unified contacts, messages) +3. **Real-World Bridge**: Connect to real smartphones via WebRTC for mixed-reality communication +4. **Virtual Hardware**: Game engine provides camera textures, audio, and other "hardware" to the phone OS +5. **Store Ecosystem**: Downloadable apps from a store, self-contained packages with isolation + +### Integration Points + +| Platform | Location | Status | +|----------|----------|--------| +| Unity Plugin | `D:\Dev\Mosis\Mosis Unity` | Binder client, raycast, viewport | +| Unreal Plugin | `D:\Dev\Mosis\Mosis Unreal` | WIP | +| Desktop Designer | `MosisService/designer/` | Complete | +| Designer Tests | `MosisService/designer-test/` | Mostly complete | + +--- + +## Milestones Overview + +| # | Milestone | Status | Description | +|---|-----------|--------|-------------| +| 1 | Cross-Platform Kernel | ✅ Complete | Desktop designer with shared kernel code | +| 2 | Testing Framework | 🔶 80% | Automated UI testing and inspection | +| 3 | Virtual Hardware | ❌ Not started | Camera, mic, speaker, filesystem APIs | +| 4 | App Sandboxing | ❌ Not started | Lua/WASM runtime, package format | +| 5 | WebRTC Bridge | ❌ Not started | Phone-to-phone communication | +| 6 | System Apps | 🔶 75% | Core phone apps | +| 7 | Game Integration | ❌ Not started | Unity/Unreal plugin polish | + +--- + +## Milestone 1: Cross-Platform Kernel ✅ COMPLETE + +**Goal**: Desktop designer with shared kernel code for rapid UI iteration. + +### Completed Tasks + +- [x] Platform abstraction layer (`src/main/kernel/include/`) + - `platform.h` - IPlatform, IGraphicsContext, IRenderTarget + - `service_interface.h` - IKernel, IServiceListener + - `file_interface.h` - IFileInterface extending Rml::FileInterface +- [x] Desktop designer project (`designer/`) + - GLFW + OpenGL 3.3 context + - Hot-reload on RML/RCSS/Lua changes + - Mouse input maps to touch events +- [x] UI designs imported from MosisDesigner + - Theme and component stylesheets + - Navigation system (Lua) + - Fonts and icons +- [x] Android build unchanged and working + +--- + +## Milestone 2: Testing Automation 🔶 80% COMPLETE + +**Goal**: Automated UI testing for rapid iteration and AI agent verification. + +### Completed Tasks + +- [x] UI hierarchy dumping to JSON +- [x] PNG screenshot capture (F12 key) +- [x] Agent-compatible JSON test results +- [x] `designer-test` executable + - WindowController (Windows SendInput API) + - HierarchyReader (element lookup by ID/class) + - LogParser (navigation event verification) +- [x] All 5 navigation tests passing +- [x] Android event injection via ADB broadcast + +### Remaining Tasks + +- [ ] **Action Recording** + - Capture tap, swipe, long_press to JSON + - Record timestamps for replay timing + - Save to `test-recordings/*.json` + +- [ ] **Action Playback** + - Load recorded JSON + - Replay with timing + - Capture results at each step + +- [ ] **Screenshot Diff** + - Compare two PNG screenshots + - Highlight pixel differences + - Report diff percentage + +### Test Recording Format + +```json +{ + "name": "Navigate to contacts and back", + "recorded_at": "2024-01-16T12:00:00Z", + "resolution": {"width": 677, "height": 1202}, + "actions": [ + {"type": "tap", "x": 413, "y": 1174, "timestamp": 0}, + {"type": "wait", "duration": 1000, "timestamp": 100}, + {"type": "tap", "x": 40, "y": 28, "timestamp": 1100}, + {"type": "wait", "duration": 500, "timestamp": 1200} + ], + "assertions": [ + {"after_action": 0, "type": "screen_is", "expected": "contacts"}, + {"after_action": 2, "type": "screen_is", "expected": "home"} + ] +} +``` + +--- + +## Milestone 3: Virtual Hardware ❌ NOT STARTED + +**Goal**: Hardware-like APIs backed by game engine or real devices. + +### 3.1 Camera Interface + +```cpp +class ICamera { + virtual void RequestFrame(FrameCallback callback) = 0; + virtual void SetResolution(int width, int height) = 0; + virtual bool IsAvailable() const = 0; +}; +``` + +**Implementations**: +- **Game Mode**: Receives texture from Unity/Unreal +- **Desktop Mode**: System webcam (optional) +- **Android Test Mode**: SharedTexture from MainActivity +- **Mock Mode**: Test patterns + +### 3.2 Microphone Interface + +```cpp +class IMicrophone { + virtual void StartCapture(AudioCallback callback) = 0; + virtual void StopCapture() = 0; + virtual bool IsAvailable() const = 0; +}; +``` + +### 3.3 Speaker Interface + +```cpp +class ISpeaker { + virtual void PlayAudio(AudioBuffer buffer) = 0; + virtual void SetVolume(float volume) = 0; +}; +``` + +### 3.4 Filesystem Interface + +```cpp +class IFileSystem { + virtual FileHandle Open(const std::string& path, Mode mode) = 0; + virtual std::vector List(const std::string& dir) = 0; + virtual bool CreateDirectory(const std::string& path) = 0; + virtual bool Delete(const std::string& path) = 0; +}; +``` + +### 3.5 Network Interface + +```cpp +class INetwork { + virtual HttpResponse Fetch(const HttpRequest& request) = 0; + virtual WebSocket Connect(const std::string& url) = 0; + virtual bool IsOnline() const = 0; +}; +``` + +--- + +## Milestone 4: App Sandboxing ❌ NOT STARTED + +**Goal**: Secure app runtime with defined package format. + +### 4.1 Runtime Decision + +| Aspect | Lua | WASM | +|--------|-----|------| +| Isolation | Weak | Strong | +| Performance | Good for UI | Near-native | +| RmlUi Integration | Native | Needs bridge | +| Ecosystem | Small | Large | + +**Recommendation**: Hybrid approach +- Lua for UI scripting (RmlUi integration) +- WASM for app logic needing isolation (future) + +### 4.2 Package Format (.mpkg) + +``` +myapp.mpkg/ +├── manifest.json # Metadata, permissions, entry +├── ui/ +│ ├── main.rml +│ ├── styles.rcss +│ └── scripts/ +├── assets/ +│ ├── icons/ +│ └── images/ +└── data/ +``` + +### 4.3 Manifest Schema + +```json +{ + "id": "com.example.myapp", + "name": "My App", + "version": "1.0.0", + "permissions": ["camera", "network", "storage"], + "entry": "ui/main.rml", + "icon": "assets/icons/app.png" +} +``` + +### 4.4 App Lifecycle + +``` +INSTALLED → LAUNCHING → RUNNING → PAUSED → STOPPED → UNINSTALLED +``` + +--- + +## Milestone 5: WebRTC Bridge ❌ NOT STARTED + +**Goal**: Cross-device communication via WebRTC. + +### 5.1 Dependencies + +- libdatachannel (add to vcpkg) + +### 5.2 Communication Protocol + +```json +{ + "type": "call" | "message" | "file" | "screen_share", + "from": "phone_id", + "to": "phone_id", + "payload": { ... } +} +``` + +### 5.3 Use Cases + +- Voice/video calls between virtual phones +- Text messaging across games +- File sharing +- Screen sharing + +### 5.4 Components + +- **WebRTCBridge** class for data channels +- **Signaling Server** (WebSocket-based) +- **Real Smartphone Bridge** (companion app) + +--- + +## Milestone 6: System Apps 🔶 75% COMPLETE + +### Completed Apps + +| App | Features | Status | +|-----|----------|--------| +| Home | App grid, dock, navigation | ✅ Complete | +| Dialer | Keypad, call UI (mock) | ✅ Complete | +| Messages | Conversation list, chat view | ✅ Complete | +| Contacts | Contact list, search, detail | ✅ Complete | +| Settings | Display, sound, about | ✅ Complete | +| Browser | URL bar, placeholder | ✅ Complete | + +### Remaining Apps + +| App | Features | Status | +|-----|----------|--------| +| Store | Browse apps, install packages | ❌ UI only | +| Camera | Viewfinder, capture, gallery | ❌ UI only | +| Music/Audio | Playback, playlists | ❌ Outline only | + +### App Data Persistence (Future) + +- Contacts: JSON or SQLite storage +- Messages: Conversation history +- Settings: Preferences file +- Photos: Virtual filesystem + +--- + +## Milestone 7: Game Integration ❌ NOT STARTED + +**Goal**: Production-ready game engine plugins. + +### 7.1 Unity Package + +- [ ] Improved raycast interaction +- [ ] Hardware button simulation (volume, power) +- [ ] Virtual camera provider (RenderTexture → phone) +- [ ] Virtual microphone provider +- [ ] Virtual speaker provider + +### 7.2 Unreal Plugin + +- [ ] Full Binder client implementation +- [ ] Blueprint integration +- [ ] Same features as Unity + +### 7.3 Documentation + +- [ ] How to embed phone in game scene +- [ ] How to provide virtual hardware +- [ ] How to handle phone events + +--- + +## Extended Features (Future) + +| Feature | Description | +|---------|-------------| +| Notifications | System notification center, per-app channels | +| Widgets | Home screen widgets, live data | +| Gestures | Swipe up for home, swipe down for notifications | +| Themes | Dark/light mode, custom colors | +| Accessibility | Font size, high contrast, TTS | +| Multi-Window | Split-screen, picture-in-picture | +| Clipboard | Copy/paste between apps and game | +| Voice Assistant | Wake word, voice commands | +| Virtual Location | GPS from game, map integration | + +--- + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ VR Game (Unity/Unreal) │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ +│ │ Phone Render │ │ Touch/Button │ │ Virtual Hardware Provider │ │ +│ │ Viewport │ │ Input │ │ (Camera, Mic, Speakers) │ │ +│ └──────┬───────┘ └──────┬───────┘ └────────────┬─────────────┘ │ +└─────────┼─────────────────┼────────────────────────┼────────────────┘ + │ Binder IPC │ Binder IPC │ Binder IPC + ▼ ▼ ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Mosis Service │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ Kernel │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │ +│ │ │ RmlUi │ │ App Runtime │ │ Virtual Hardware │ │ │ +│ │ │ Renderer │ │ (Lua) │ │ Abstraction │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │ +│ │ │ App │ │ IPC │ │ libdatachannel │ │ │ +│ │ │ Manager │ │ Router │ │ (WebRTC Bridge) │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + │ + │ WebRTC (libdatachannel) + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Real Smartphone / Other Devices │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Folder Structure + +``` +MosisService/ +├── src/main/ +│ ├── kernel/ # Shared cross-platform kernel +│ │ ├── include/ # Platform abstractions +│ │ └── src/ # Shared implementation +│ ├── cpp/ # Android-specific native code +│ ├── java/ # Kotlin/Java code +│ └── assets/ # Shared UI assets +│ ├── apps/ # System apps +│ │ ├── home/ +│ │ ├── dialer/ +│ │ ├── messages/ +│ │ ├── contacts/ +│ │ ├── settings/ +│ │ ├── browser/ +│ │ ├── store/ # TODO +│ │ └── camera/ # TODO +│ ├── ui/ # Shared stylesheets +│ ├── scripts/ # Lua scripts +│ ├── icons/ # TGA icons +│ └── fonts/ # TTF fonts +│ +├── designer/ # Desktop designer +│ ├── src/ +│ │ ├── testing/ # UI inspection, capture +│ │ └── ... +│ └── build/ +│ +├── designer-test/ # Automated UI tests +│ ├── src/ +│ └── build/ +│ +└── docs/ # Documentation (future) +``` + +--- + +## Build Commands + +```bash +# Android +./gradlew assembleDebug +./gradlew installDebug + +# Desktop Designer +cd designer +cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake +cmake --build build --config Debug +./build/Debug/mosis-designer.exe ../src/main/assets/apps/home/home.rml + +# Designer Tests +cd designer-test +cmake -B build -DCMAKE_TOOLCHAIN_FILE=%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake +cmake --build build --config Debug +./build/Debug/designer-test.exe +``` + +--- + +## Current Sprint: Complete Partial Tasks + +### Priority 1: Testing Framework Completion +- [ ] Action recording (capture interactions to JSON) +- [ ] Action playback (replay with timing) +- [ ] Screenshot diff (visual regression) + +### Priority 2: Remaining System Apps +- [ ] Store app (UI only - browse, install) +- [ ] Camera app (UI + shared texture from MainActivity) +- [ ] Music app (UI outline only) + +### Priority 3: App Data Persistence +- [ ] JSON/SQLite storage layer +- [ ] Contact CRUD operations +- [ ] Message history +- [ ] Settings persistence + +--- + +## Resources + +### Material Design + +- **Icons**: `D:\Dev\Mosis\MosisDesigner\material-design-icons` + - 20 categories, SVG/PNG/Font formats + - Browse at https://fonts.google.com/icons + +- **Components**: `D:\Dev\Mosis\MosisDesigner\material-design-lite` + - Reference implementation for design patterns + +### Documentation + +- `CLAUDE.md` - Development guide +- `TESTING.md` - Testing framework documentation +- `ROADMAP.md` - This file + +--- + +*Last updated: 2024-01-16* diff --git a/src/main/assets/apps/camera/camera.rml b/src/main/assets/apps/camera/camera.rml new file mode 100644 index 0000000..4a4aa89 --- /dev/null +++ b/src/main/assets/apps/camera/camera.rml @@ -0,0 +1,385 @@ + + + + + + + Camera + + + + +
+
+ +
+
+
+ +
+
+ +
+
+ +
+
+
+ + +
+
+ +
+
C
+
Camera Preview
+
+ Tap to focus +
+
+ + +
+
+
+
+
+
+ + +
+
+ + +
Flash: Auto
+ + +
Timer: Off
+ + +
+
-
+ 1.0x +
+
+
+
+ + +
+ Night + Portrait + Photo + Video + More +
+ + +
+
+ +
+
+
+
+
+ +
+
+ + diff --git a/src/main/assets/apps/home/home.rml b/src/main/assets/apps/home/home.rml index f7237cd..257c1cc 100644 --- a/src/main/assets/apps/home/home.rml +++ b/src/main/assets/apps/home/home.rml @@ -73,16 +73,16 @@
Gallery -
-
+
+
Camera
Settings
-
-
+
+
Music
@@ -105,8 +105,8 @@
-
-
+
+
Store
diff --git a/src/main/assets/apps/music/music.rml b/src/main/assets/apps/music/music.rml new file mode 100644 index 0000000..49780bd --- /dev/null +++ b/src/main/assets/apps/music/music.rml @@ -0,0 +1,410 @@ + + + + + + + Music + + + + +
+
+ +
+ Music +
+ +
+
+ + +
+ +
+ Good afternoon +
+ + +
+
+
L
+ Liked Songs +
+
+
D
+ Daily Mix 1 +
+
+
R
+ Release Radar +
+
+
C
+ Chill Vibes +
+
+
W
+ Workout Mix +
+
+
F
+ Focus Flow +
+
+ + +
+ Recently Played + SEE ALL +
+ +
+
+
P
+
Pop Hits
+
Playlist
+
+
+
E
+
Electronic
+
Playlist
+
+
+
J
+
Jazz Classics
+
Playlist
+
+
+
R
+
Rock Legends
+
Playlist
+
+
+ + +
+ Made For You + SEE ALL +
+ +
+
1
+
+
Daily Mix 1
+
Based on your listening
+
+
+ +
+
2
+
+
Daily Mix 2
+
Electronic, Ambient, Chill
+
+
+ +
+
D
+
+
Discover Weekly
+
Your weekly mixtape
+
+
+ +
+
R
+
+
Release Radar
+
New music from artists you follow
+
+
+
+ + +
+
M
+
+
Midnight City
+
M83
+
+
+
+ +
+
+ +
+
+
+ + +
+
+ + Home +
+
+ + Search +
+
+ + Library +
+
+ + diff --git a/src/main/assets/apps/store/store.rml b/src/main/assets/apps/store/store.rml new file mode 100644 index 0000000..f3dad3b --- /dev/null +++ b/src/main/assets/apps/store/store.rml @@ -0,0 +1,488 @@ + + + + + + + Store + + + + +
+
+ +
+ Mosis Store +
+ +
+
+ + +
+ +
+ + Search apps & games +
+ + +
+ Featured + Weather Pro + Beautiful forecasts for your virtual world +
+ + +
+
For You
+
Games
+
Social
+
Productivity
+
Entertainment
+
Tools
+
+ + +
+ Recommended for You + See all +
+ +
+
+
N
+
Notes
+
Productivity
+
4.7
+
+
+
C
+
Calculator
+
Tools
+
4.5
+
+
+
W
+
Weather
+
Weather
+
4.8
+
+
+
M
+
Maps
+
Navigation
+
4.6
+
+
+ + +
+ Top Free Apps + See all +
+ + +
+
S
+
+
Social Hub
+
Social • 12 MB
+
+ 4.9 • 1.2M downloads +
+
+
Install
+
+ +
+
G
+
+
Games Center
+
Games • 45 MB
+
+ 4.7 • 890K downloads +
+
+
Install
+
+ +
+
F
+
+
File Manager
+
Tools • 8 MB
+
+ 4.6 • 650K downloads +
+
+
Open
+
+ +
+
M
+
+
Music Player
+
Music • 18 MB
+
+ 4.5 • 520K downloads +
+
+
Install
+
+ +
+
P
+
+
Photo Editor
+
Photography • 32 MB
+
+ 4.4 • 410K downloads +
+
+
Install
+
+ + +
+ New Games + See all +
+ +
+
+
P
+
Puzzle Quest
+
Puzzle
+
4.8
+
+
+
R
+
Racing VR
+
Racing
+
4.6
+
+
+
S
+
Space Explorer
+
Adventure
+
4.7
+
+
+
C
+
Card Master
+
Card
+
4.5
+
+
+
+ + +
+
+ + Apps +
+
+ + Games +
+
+ + Updates +
+
+ + diff --git a/src/main/assets/icons/account.tga b/src/main/assets/icons/account.tga new file mode 100644 index 0000000..71e7f88 --- /dev/null +++ b/src/main/assets/icons/account.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7d06b26a0c072f57aef0ffaa4fc918766e6c6526213ca799d1e6d260be2a275e +size 9234 diff --git a/src/main/assets/icons/download.tga b/src/main/assets/icons/download.tga new file mode 100644 index 0000000..646d3f6 --- /dev/null +++ b/src/main/assets/icons/download.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f6ba618cfd901d45f2cc4ca1ca7951d489f75640db68bcbd49b7b44e27b9f5e2 +size 9234 diff --git a/src/main/assets/icons/flash.tga b/src/main/assets/icons/flash.tga new file mode 100644 index 0000000..691b0e0 --- /dev/null +++ b/src/main/assets/icons/flash.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7b4f073aff4a6c7a3b553a19b8a9481344b97c356cf6d198d949d8e548942739 +size 9234 diff --git a/src/main/assets/icons/game.tga b/src/main/assets/icons/game.tga new file mode 100644 index 0000000..93f5ecd --- /dev/null +++ b/src/main/assets/icons/game.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ea10844fa645a317801194d59d071d74563f1d561949f7f3ad7b4ab2fb935549 +size 9234 diff --git a/src/main/assets/icons/heart.tga b/src/main/assets/icons/heart.tga new file mode 100644 index 0000000..da61902 --- /dev/null +++ b/src/main/assets/icons/heart.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eef94f67fd12188585fb3776220d72e976146f4186b0da38f53e09632bde61c3 +size 9234 diff --git a/src/main/assets/icons/library.tga b/src/main/assets/icons/library.tga new file mode 100644 index 0000000..93bfb86 --- /dev/null +++ b/src/main/assets/icons/library.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b9f901748c8b65a67692fb4c18f6f04974191c25fb22685e0b7fe4b0b9101d9e +size 9234 diff --git a/src/main/assets/icons/play.tga b/src/main/assets/icons/play.tga new file mode 100644 index 0000000..2e0cfd0 --- /dev/null +++ b/src/main/assets/icons/play.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:007531183043cdb419b3d3c95c2032f4895c1958df008c670ed5b476dbb92737 +size 9234 diff --git a/src/main/assets/icons/switch-camera.tga b/src/main/assets/icons/switch-camera.tga new file mode 100644 index 0000000..1b6e09a --- /dev/null +++ b/src/main/assets/icons/switch-camera.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:414ac737296e8da98f0b7fd04758919e89239037641ef0382b80a3df75a30a15 +size 9234 diff --git a/src/main/assets/icons/timer.tga b/src/main/assets/icons/timer.tga new file mode 100644 index 0000000..3d3c501 --- /dev/null +++ b/src/main/assets/icons/timer.tga @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5a60b7132b69e4747a8071404c5b668ba641ae56d7a02c721cb8a8ecc7d3b591 +size 9234 diff --git a/src/main/assets/scripts/navigation.lua b/src/main/assets/scripts/navigation.lua index e004ed7..f5eb9ef 100644 --- a/src/main/assets/scripts/navigation.lua +++ b/src/main/assets/scripts/navigation.lua @@ -12,7 +12,10 @@ local screens = { messages = "apps/messages/messages.rml", chat = "apps/messages/chat.rml", settings = "apps/settings/settings.rml", - browser = "apps/browser/browser.rml" + browser = "apps/browser/browser.rml", + store = "apps/store/store.rml", + camera = "apps/camera/camera.rml", + music = "apps/music/music.rml" } -- Use global state to persist across document loads