MosisVR/Packages/com.omarator.mosissdk

MosisSDK for Unity

Unity package providing integration with MosisService for virtual smartphone display in VR applications.

Overview

This package connects to the MosisService Android app via AIDL Binder IPC to:

• Receive rendered phone screen frames via AHardwareBuffer
• Forward VR controller touch input to the virtual phone
• Support both OpenGL ES and Vulkan graphics backends

Requirements

• Unity 6000.3.2f1 or later
• Android target platform
• MosisService app installed on device

Installation

1. Open Window > Package Manager
2. Click + > Add package from disk
3. Navigate to Packages/com.omarator.mosissdk/package.json

Project Structure

Packages/com.omarator.mosissdk/
├── package.json
├── README.md
├── Runtime/
│   ├── KotlinBridge.cs          # C# to native bridge
│   └── com.omarator.mosissdk.asmdef
├── Plugins/
│   └── Android/
│       ├── MosisSDK.aar         # Kotlin service connection
│       ├── libs/
│       │   └── arm64-v8a/
│       │       └── libmy_native_lib.so
│       └── cpp/                 # Native source
│           ├── CMakeLists.txt
│           ├── my_native_code.cpp
│           ├── texture_backend.h
│           ├── opengl_backend.h/cpp
│           └── vulkan_backend.h/cpp
└── Assets/
    └── PhoneInteraction.cs      # VR controller input

Build Commands

Option 1: Direct APK Build (Recommended)

Build directly to APK without exporting:

"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidDirectCI ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk" ^
    -logFile "D:\Dev\Mosis\Builds\Unity\build.log"

Or from Unity Editor: Build > Build Android APK (Direct)

Option 2: Export + Gradle Build

Export a Gradle project for more control over the build process:

Step 1: Export from Unity

"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidCI ^
    -export true ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR" ^
    -logFile "D:\Dev\Mosis\Builds\Unity\build.log"

Or from Unity Editor: Build > Export Android Project

Step 2: Build with Gradle

cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
gradle assembleRelease

:: APK will be at:
:: launcher\build\outputs\apk\release\launcher-release.apk

Or open in Android Studio and build from there.

Unity Editor Manual Build

1. File > Build Settings
2. Switch Platform to Android
3. Player Settings > Other Settings:
   • Scripting Backend: IL2CPP
   • Target Architectures: ARM64
   • Graphics APIs: Vulkan, OpenGLES3 (in order of preference)
   • Minimum API Level: 29
4. For direct APK: Uncheck "Export Project", click Build
5. For export: Check "Export Project", click Export

Native Plugin Build (Manual)

The native plugin is built automatically during Unity's build process via CMake. To rebuild manually:

cd Packages/com.omarator.mosissdk/Plugins/Android/cpp

:: Configure with Android NDK
cmake -B build ^
    -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
    -DANDROID_ABI=arm64-v8a ^
    -DANDROID_PLATFORM=android-29

:: Build
cmake --build build

:: Copy output
copy build\libmy_native_lib.so ..\libs\arm64-v8a\

Usage

Basic Setup

1. Add KotlinBridge component to a GameObject in your scene
2. Add PhoneInteraction component for VR input handling
3. Assign VR controller and phone plane references

KotlinBridge

The KotlinBridge component manages the connection to MosisService:

public class KotlinBridge : MonoBehaviour
{
    public Material phoneMaterial;  // Material to apply phone texture

    // Called automatically when texture is ready
    // Texture is applied to phoneMaterial's _MainTex
}

PhoneInteraction

Handles VR controller raycasting and touch input:

public class PhoneInteraction : MonoBehaviour
{
    public GameObject Controller;       // VR controller
    public GameObject Plane;            // Phone display plane
    public XRNode inputDevice;          // Controller hand
    public float triggerThreshold;      // Trigger press threshold
}

Touch API

Send touch events from custom scripts:

// Touch down at normalized UV (0-1)
KotlinBridge.SendTouchDown(0.5f, 0.5f);

// Touch move
KotlinBridge.SendTouchMove(0.6f, 0.5f);

// Touch up
KotlinBridge.SendTouchUp(0.6f, 0.5f);

Graphics Backend

The native plugin automatically selects the appropriate backend:

Graphics API	Backend	Status
Vulkan	VulkanTextureBackend	Preferred
OpenGL ES 3.0	OpenGLTextureBackend	Fallback

Backend selection happens in UnityPluginLoad() based on the active renderer.

Note: Unity 6 requires OpenGL ES 3.0 minimum. OpenGL ES 2.0 is not supported.

Vulkan Import

When using Vulkan, the plugin imports AHardwareBuffer directly as a VkImage:

1. Query buffer properties via vkGetAndroidHardwareBufferPropertiesANDROID
2. Create VkImage with external memory handle type
3. Import memory with VkImportAndroidHardwareBufferInfoANDROID
4. Copy to local image each frame for safe rendering

OpenGL Import

When using OpenGL ES, the plugin uses EGL image extension:

1. Create EGLImage from AHardwareBuffer via eglCreateImageKHR
2. Bind to OpenGL texture via glEGLImageTargetTexture2DOES
3. Blit to local texture each frame

Device Testing

# Install MosisService first
adb install -r MosisService-debug.apk

# Install Unity app
adb install -r MosisVR.apk

# Launch service
adb shell am start -n com.omixlab.mosis/.MainActivity

# Launch Unity app
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity

# Monitor logs
adb logcat -s Unity MosisSDK Vulkan

Troubleshooting

Black Screen / No Texture

• Verify MosisService is running (adb shell ps | grep mosis)
• Check logcat for connection errors
• Ensure both apps have same user ID or are debuggable

Vulkan Errors

• Device must support VK_ANDROID_external_memory_android_hardware_buffer
• Check adb logcat -s Vulkan for extension availability
• Falls back to OpenGL if Vulkan init fails

Touch Not Working

• Verify PhoneInteraction references are set
• Check XR input device selection matches controller
• Test with mouse click fallback in editor

Build Fails with "glad/gles2.h not found"

This error occurs when IL2CPP tries to compile native plugin C++ files. The .meta files for the cpp/h files must have PluginImporter settings with enabled: 0 for all platforms to exclude them from IL2CPP.

Solution: Ensure all .meta files in Plugins/Android/cpp/ have proper PluginImporter exclusion settings:

PluginImporter:
  platformData:
    Android:
      enabled: 0
    Any:
      enabled: 0
      settings:
        Exclude Android: 1
        Exclude Editor: 1
        # ... exclude all platforms

Build Fails with "getSdkDir() not found"

The mainTemplate.gradle must use System.getenv("ANDROID_HOME") instead of getSdkDir() for the ANDROID_SDK cmake argument. Ensure ANDROID_HOME environment variable is set.

Implementation Notes

Files Modified in Vulkan Implementation

File	Changes
my_native_code.cpp	Backend abstraction, UnityPluginLoad detection
CMakeLists.txt	Added Vulkan library, new source files
KotlinBridge.cs	Added SendTouchDown/SendTouchUp, dynamic resolution
PhoneInteraction.cs	Added VR trigger-based touch handling

New Files Added

File	Purpose
texture_backend.h	ITextureBackend abstract interface
opengl_backend.h/cpp	OpenGL ES implementation
vulkan_backend.h/cpp	Vulkan implementation

Version History

• 1.0.0 - Initial release with OpenGL support
• 1.1.0 - Added Vulkan backend, touch down/up events, dynamic resolution
• 1.2.0 - Fixed IL2CPP build compatibility, added direct APK build support, fixed mainTemplate.gradle for direct builds