MosisVR/Packages/com.omarator.mosissdk/README.md

# MosisSDK for Unity

Unity package providing integration with MosisService for virtual smartphone display in VR applications.

## Overview

This package connects to the MosisService Android app via AIDL Binder IPC to:
- Receive rendered phone screen frames via AHardwareBuffer
- Forward VR controller touch input to the virtual phone
- Support both OpenGL ES and Vulkan graphics backends

## Requirements

- Unity 6000.3.2f1 or later
- Android target platform
- MosisService app installed on device

## Installation

1. Open Window > Package Manager
2. Click + > Add package from disk
3. Navigate to `Packages/com.omarator.mosissdk/package.json`

## Project Structure

```
Packages/com.omarator.mosissdk/
├── package.json
├── README.md
├── Runtime/
│   ├── KotlinBridge.cs          # C# to native bridge
│   └── com.omarator.mosissdk.asmdef
├── Plugins/
│   └── Android/
│       ├── MosisSDK.aar         # Kotlin service connection
│       ├── libs/
│       │   └── arm64-v8a/
│       │       └── libmy_native_lib.so
│       └── cpp/                 # Native source
│           ├── CMakeLists.txt
│           ├── my_native_code.cpp
│           ├── texture_backend.h
│           ├── opengl_backend.h/cpp
│           └── vulkan_backend.h/cpp
└── Assets/
    └── PhoneInteraction.cs      # VR controller input
```

## Build Commands

### Option 1: Direct APK Build (Recommended)

Build directly to APK without exporting:

```batch
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidDirectCI ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk" ^
    -logFile "D:\Dev\Mosis\Builds\Unity\build.log"
```

Or from Unity Editor: **Build > Build Android APK (Direct)**

### Option 2: Export + Gradle Build

Export a Gradle project for more control over the build process:

**Step 1: Export from Unity**
```batch
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidCI ^
    -export true ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR" ^
    -logFile "D:\Dev\Mosis\Builds\Unity\build.log"
```

Or from Unity Editor: **Build > Export Android Project**

**Step 2: Build with Gradle**
```batch
cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
gradle assembleRelease

:: APK will be at:
:: launcher\build\outputs\apk\release\launcher-release.apk
```

Or open in Android Studio and build from there.

### Unity Editor Manual Build

1. File > Build Settings
2. Switch Platform to Android
3. Player Settings > Other Settings:
   - Scripting Backend: IL2CPP
   - Target Architectures: ARM64
   - Graphics APIs: Vulkan, OpenGLES3 (in order of preference)
   - Minimum API Level: 29
4. For direct APK: Uncheck "Export Project", click Build
5. For export: Check "Export Project", click Export

### Native Plugin Build (Manual)

The native plugin is built automatically during Unity's build process via CMake. To rebuild manually:

```batch
cd Packages/com.omarator.mosissdk/Plugins/Android/cpp

:: Configure with Android NDK
cmake -B build ^
    -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
    -DANDROID_ABI=arm64-v8a ^
    -DANDROID_PLATFORM=android-29

:: Build
cmake --build build

:: Copy output
copy build\libmy_native_lib.so ..\libs\arm64-v8a\
```

## Usage

### Basic Setup

1. Add `KotlinBridge` component to a GameObject in your scene
2. Add `PhoneInteraction` component for VR input handling
3. Assign VR controller and phone plane references

### KotlinBridge

The `KotlinBridge` component manages the connection to MosisService:

```csharp
public class KotlinBridge : MonoBehaviour
{
    public Material phoneMaterial;  // Material to apply phone texture

    // Called automatically when texture is ready
    // Texture is applied to phoneMaterial's _MainTex
}
```

### PhoneInteraction

Handles VR controller raycasting and touch input:

```csharp
public class PhoneInteraction : MonoBehaviour
{
    public GameObject Controller;       // VR controller
    public GameObject Plane;            // Phone display plane
    public XRNode inputDevice;          // Controller hand
    public float triggerThreshold;      // Trigger press threshold
}
```

### Touch API

Send touch events from custom scripts:

```csharp
// Touch down at normalized UV (0-1)
KotlinBridge.SendTouchDown(0.5f, 0.5f);

// Touch move
KotlinBridge.SendTouchMove(0.6f, 0.5f);

// Touch up
KotlinBridge.SendTouchUp(0.6f, 0.5f);
```

## Graphics Backend

The native plugin automatically selects the appropriate backend:

| Graphics API | Backend | Status |
|--------------|---------|--------|
| Vulkan | VulkanTextureBackend | Preferred |
| OpenGL ES 3.0 | OpenGLTextureBackend | Fallback |

Backend selection happens in `UnityPluginLoad()` based on the active renderer.

Note: Unity 6 requires OpenGL ES 3.0 minimum. OpenGL ES 2.0 is not supported.

### Vulkan Import

When using Vulkan, the plugin imports AHardwareBuffer directly as a VkImage:

1. Query buffer properties via `vkGetAndroidHardwareBufferPropertiesANDROID`
2. Create VkImage with external memory handle type
3. Import memory with `VkImportAndroidHardwareBufferInfoANDROID`
4. Copy to local image each frame for safe rendering

### OpenGL Import

When using OpenGL ES, the plugin uses EGL image extension:

1. Create EGLImage from AHardwareBuffer via `eglCreateImageKHR`
2. Bind to OpenGL texture via `glEGLImageTargetTexture2DOES`
3. Blit to local texture each frame

## Device Testing

```bash
# Install MosisService first
adb install -r MosisService-debug.apk

# Install Unity app
adb install -r MosisVR.apk

# Launch service
adb shell am start -n com.omixlab.mosis/.MainActivity

# Launch Unity app
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity

# Monitor logs
adb logcat -s Unity MosisSDK Vulkan
```

## Troubleshooting

### Black Screen / No Texture

- Verify MosisService is running (`adb shell ps | grep mosis`)
- Check logcat for connection errors
- Ensure both apps have same user ID or are debuggable

### Vulkan Errors

- Device must support `VK_ANDROID_external_memory_android_hardware_buffer`
- Check `adb logcat -s Vulkan` for extension availability
- Falls back to OpenGL if Vulkan init fails

### Touch Not Working

- Verify `PhoneInteraction` references are set
- Check XR input device selection matches controller
- Test with mouse click fallback in editor

### Build Fails with "glad/gles2.h not found"

This error occurs when IL2CPP tries to compile native plugin C++ files. The `.meta` files for the cpp/h files must have `PluginImporter` settings with `enabled: 0` for all platforms to exclude them from IL2CPP.

**Solution**: Ensure all `.meta` files in `Plugins/Android/cpp/` have proper PluginImporter exclusion settings:
```yaml
PluginImporter:
  platformData:
    Android:
      enabled: 0
    Any:
      enabled: 0
      settings:
        Exclude Android: 1
        Exclude Editor: 1
        # ... exclude all platforms
```

### Build Fails with "getSdkDir() not found"

The `mainTemplate.gradle` must use `System.getenv("ANDROID_HOME")` instead of `getSdkDir()` for the ANDROID_SDK cmake argument. Ensure `ANDROID_HOME` environment variable is set.

## Implementation Notes

### Files Modified in Vulkan Implementation

| File | Changes |
|------|---------|
| `my_native_code.cpp` | Backend abstraction, `UnityPluginLoad` detection |
| `CMakeLists.txt` | Added Vulkan library, new source files |
| `KotlinBridge.cs` | Added `SendTouchDown`/`SendTouchUp`, dynamic resolution |
| `PhoneInteraction.cs` | Added VR trigger-based touch handling |

### New Files Added

| File | Purpose |
|------|---------|
| `texture_backend.h` | ITextureBackend abstract interface |
| `opengl_backend.h/cpp` | OpenGL ES implementation |
| `vulkan_backend.h/cpp` | Vulkan implementation |

## Version History

- **1.0.0** - Initial release with OpenGL support
- **1.1.0** - Added Vulkan backend, touch down/up events, dynamic resolution
- **1.2.0** - Fixed IL2CPP build compatibility, added direct APK build support, fixed mainTemplate.gradle for direct builds