MosisVR/Packages/com.omarator.mosissdk/README.md

MosisSDK for Unity

Unity package providing integration with MosisService for virtual smartphone display in VR applications.

Overview

This package connects to the MosisService Android app via AIDL Binder IPC to:

• Receive rendered phone screen frames via AHardwareBuffer
• Forward VR controller touch input to the virtual phone
• Support both OpenGL ES and Vulkan graphics backends

Requirements

• Unity 6000.3.2f1 or later
• Android target platform
• MosisService app installed on device

Installation

1. Open Window > Package Manager
2. Click + > Add package from disk
3. Navigate to Packages/com.omarator.mosissdk/package.json

Project Structure

Packages/com.omarator.mosissdk/
├── package.json
├── README.md
├── Runtime/
│   ├── KotlinBridge.cs          # C# to native bridge
│   └── com.omarator.mosissdk.asmdef
├── Plugins/
│   └── Android/
│       ├── MosisSDK.aar         # Kotlin service connection
│       ├── libs/
│       │   └── arm64-v8a/
│       │       └── libmy_native_lib.so
│       └── cpp/                 # Native source
│           ├── CMakeLists.txt
│           ├── my_native_code.cpp
│           ├── texture_backend.h
│           ├── opengl_backend.h/cpp
│           └── vulkan_backend.h/cpp
└── Assets/
    └── PhoneInteraction.cs      # VR controller input

Build Commands

Unity Build

1. File > Build Settings
2. Switch Platform to Android
3. Player Settings > Other Settings:
   • Graphics APIs: Vulkan, OpenGLES3 (in order of preference)
   • Minimum API Level: 29
4. Build or Build and Run

Native Plugin Build

To rebuild the native library manually:

cd Packages/com.omarator.mosissdk/Plugins/Android/cpp

:: Configure with Android NDK
cmake -B build ^
    -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
    -DANDROID_ABI=arm64-v8a ^
    -DANDROID_PLATFORM=android-29

:: Build
cmake --build build

:: Copy output
copy build\libmy_native_lib.so ..\libs\arm64-v8a\

Usage

Basic Setup

1. Add KotlinBridge component to a GameObject in your scene
2. Add PhoneInteraction component for VR input handling
3. Assign VR controller and phone plane references

KotlinBridge

The KotlinBridge component manages the connection to MosisService:

public class KotlinBridge : MonoBehaviour
{
    public Material phoneMaterial;  // Material to apply phone texture

    // Called automatically when texture is ready
    // Texture is applied to phoneMaterial's _MainTex
}

PhoneInteraction

Handles VR controller raycasting and touch input:

public class PhoneInteraction : MonoBehaviour
{
    public GameObject Controller;       // VR controller
    public GameObject Plane;            // Phone display plane
    public XRNode inputDevice;          // Controller hand
    public float triggerThreshold;      // Trigger press threshold
}

Touch API

Send touch events from custom scripts:

// Touch down at normalized UV (0-1)
KotlinBridge.SendTouchDown(0.5f, 0.5f);

// Touch move
KotlinBridge.SendTouchMove(0.6f, 0.5f);

// Touch up
KotlinBridge.SendTouchUp(0.6f, 0.5f);

Graphics Backend

The native plugin automatically selects the appropriate backend:

Graphics API	Backend	Status
Vulkan	VulkanTextureBackend	Preferred
OpenGL ES 3.0	OpenGLTextureBackend	Fallback
OpenGL ES 2.0	OpenGLTextureBackend	Fallback

Backend selection happens in UnityPluginLoad() based on the active renderer.

Vulkan Import

When using Vulkan, the plugin imports AHardwareBuffer directly as a VkImage:

1. Query buffer properties via vkGetAndroidHardwareBufferPropertiesANDROID
2. Create VkImage with external memory handle type
3. Import memory with VkImportAndroidHardwareBufferInfoANDROID
4. Copy to local image each frame for safe rendering

OpenGL Import

When using OpenGL ES, the plugin uses EGL image extension:

1. Create EGLImage from AHardwareBuffer via eglCreateImageKHR
2. Bind to OpenGL texture via glEGLImageTargetTexture2DOES
3. Blit to local texture each frame

Device Testing

# Install MosisService first
adb install -r MosisService-debug.apk

# Install Unity app
adb install -r MosisVR.apk

# Launch service
adb shell am start -n com.omixlab.mosis/.MainActivity

# Launch Unity app
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity

# Monitor logs
adb logcat -s Unity MosisSDK Vulkan

Troubleshooting

Black Screen / No Texture

• Verify MosisService is running (adb shell ps | grep mosis)
• Check logcat for connection errors
• Ensure both apps have same user ID or are debuggable

Vulkan Errors

• Device must support VK_ANDROID_external_memory_android_hardware_buffer
• Check adb logcat -s Vulkan for extension availability
• Falls back to OpenGL if Vulkan init fails

Touch Not Working

• Verify PhoneInteraction references are set
• Check XR input device selection matches controller
• Test with mouse click fallback in editor

Implementation Notes

Files Modified in Vulkan Implementation

File	Changes
my_native_code.cpp	Backend abstraction, UnityPluginLoad detection
CMakeLists.txt	Added Vulkan library, new source files
KotlinBridge.cs	Added SendTouchDown/SendTouchUp, dynamic resolution
PhoneInteraction.cs	Added VR trigger-based touch handling

New Files Added

File	Purpose
texture_backend.h	ITextureBackend abstract interface
opengl_backend.h/cpp	OpenGL ES implementation
vulkan_backend.h/cpp	Vulkan implementation

Version History

• 1.0.0 - Initial release with OpenGL support
• 1.1.0 - Added Vulkan backend, touch down/up events, dynamic resolution