252 lines
6.7 KiB
Markdown
252 lines
6.7 KiB
Markdown
# MosisSDK for Unity
|
|
|
|
Unity package providing integration with MosisService for virtual smartphone display in VR applications.
|
|
|
|
## Overview
|
|
|
|
This package connects to the MosisService Android app via AIDL Binder IPC to:
|
|
- Receive rendered phone screen frames via AHardwareBuffer
|
|
- Forward VR controller touch input to the virtual phone
|
|
- Support both OpenGL ES and Vulkan graphics backends
|
|
|
|
## Requirements
|
|
|
|
- Unity 6000.3.2f1 or later
|
|
- Android target platform
|
|
- MosisService app installed on device
|
|
|
|
## Installation
|
|
|
|
1. Open Window > Package Manager
|
|
2. Click + > Add package from disk
|
|
3. Navigate to `Packages/com.omarator.mosissdk/package.json`
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
Packages/com.omarator.mosissdk/
|
|
├── package.json
|
|
├── README.md
|
|
├── Runtime/
|
|
│ ├── KotlinBridge.cs # C# to native bridge
|
|
│ └── com.omarator.mosissdk.asmdef
|
|
├── Plugins/
|
|
│ └── Android/
|
|
│ ├── MosisSDK.aar # Kotlin service connection
|
|
│ ├── libs/
|
|
│ │ └── arm64-v8a/
|
|
│ │ └── libmy_native_lib.so
|
|
│ └── cpp/ # Native source
|
|
│ ├── CMakeLists.txt
|
|
│ ├── my_native_code.cpp
|
|
│ ├── texture_backend.h
|
|
│ ├── opengl_backend.h/cpp
|
|
│ └── vulkan_backend.h/cpp
|
|
└── Assets/
|
|
└── PhoneInteraction.cs # VR controller input
|
|
```
|
|
|
|
## Build Commands
|
|
|
|
### Unity Build (Export to Android Studio)
|
|
|
|
Unity exports a Gradle project which is then built in Android Studio:
|
|
|
|
1. File > Build Settings
|
|
2. Switch Platform to Android
|
|
3. Player Settings > Other Settings:
|
|
- Scripting Backend: IL2CPP
|
|
- Target Architectures: ARM64
|
|
- Graphics APIs: Vulkan, OpenGLES3 (in order of preference)
|
|
- Minimum API Level: 29
|
|
4. Check "Export Project"
|
|
5. Click Export and select output folder (e.g., `D:\Dev\Mosis\Builds\Unity\Android\MosisVR`)
|
|
|
|
### Build Gradle Project
|
|
|
|
After exporting from Unity:
|
|
|
|
```batch
|
|
cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
|
|
gradle assembleRelease
|
|
|
|
:: APK will be at:
|
|
:: launcher\build\outputs\apk\release\launcher-release.apk
|
|
```
|
|
|
|
Or open in Android Studio and build from there.
|
|
|
|
### Command Line Export (Optional)
|
|
|
|
```batch
|
|
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
|
|
-batchmode -quit -nographics ^
|
|
-projectPath "D:\Dev\Mosis\MosisVR" ^
|
|
-executeMethod BuildScript.BuildAndroidCI ^
|
|
-outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR" ^
|
|
-logFile "D:\Dev\Mosis\Builds\Unity\build.log"
|
|
```
|
|
|
|
### Native Plugin Build (Manual)
|
|
|
|
To rebuild the native library manually:
|
|
|
|
```batch
|
|
cd Packages/com.omarator.mosissdk/Plugins/Android/cpp
|
|
|
|
:: Configure with Android NDK
|
|
cmake -B build ^
|
|
-DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
|
|
-DANDROID_ABI=arm64-v8a ^
|
|
-DANDROID_PLATFORM=android-29
|
|
|
|
:: Build
|
|
cmake --build build
|
|
|
|
:: Copy output
|
|
copy build\libmy_native_lib.so ..\libs\arm64-v8a\
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Basic Setup
|
|
|
|
1. Add `KotlinBridge` component to a GameObject in your scene
|
|
2. Add `PhoneInteraction` component for VR input handling
|
|
3. Assign VR controller and phone plane references
|
|
|
|
### KotlinBridge
|
|
|
|
The `KotlinBridge` component manages the connection to MosisService:
|
|
|
|
```csharp
|
|
public class KotlinBridge : MonoBehaviour
|
|
{
|
|
public Material phoneMaterial; // Material to apply phone texture
|
|
|
|
// Called automatically when texture is ready
|
|
// Texture is applied to phoneMaterial's _MainTex
|
|
}
|
|
```
|
|
|
|
### PhoneInteraction
|
|
|
|
Handles VR controller raycasting and touch input:
|
|
|
|
```csharp
|
|
public class PhoneInteraction : MonoBehaviour
|
|
{
|
|
public GameObject Controller; // VR controller
|
|
public GameObject Plane; // Phone display plane
|
|
public XRNode inputDevice; // Controller hand
|
|
public float triggerThreshold; // Trigger press threshold
|
|
}
|
|
```
|
|
|
|
### Touch API
|
|
|
|
Send touch events from custom scripts:
|
|
|
|
```csharp
|
|
// Touch down at normalized UV (0-1)
|
|
KotlinBridge.SendTouchDown(0.5f, 0.5f);
|
|
|
|
// Touch move
|
|
KotlinBridge.SendTouchMove(0.6f, 0.5f);
|
|
|
|
// Touch up
|
|
KotlinBridge.SendTouchUp(0.6f, 0.5f);
|
|
```
|
|
|
|
## Graphics Backend
|
|
|
|
The native plugin automatically selects the appropriate backend:
|
|
|
|
| Graphics API | Backend | Status |
|
|
|--------------|---------|--------|
|
|
| Vulkan | VulkanTextureBackend | Preferred |
|
|
| OpenGL ES 3.0 | OpenGLTextureBackend | Fallback |
|
|
| OpenGL ES 2.0 | OpenGLTextureBackend | Fallback |
|
|
|
|
Backend selection happens in `UnityPluginLoad()` based on the active renderer.
|
|
|
|
### Vulkan Import
|
|
|
|
When using Vulkan, the plugin imports AHardwareBuffer directly as a VkImage:
|
|
|
|
1. Query buffer properties via `vkGetAndroidHardwareBufferPropertiesANDROID`
|
|
2. Create VkImage with external memory handle type
|
|
3. Import memory with `VkImportAndroidHardwareBufferInfoANDROID`
|
|
4. Copy to local image each frame for safe rendering
|
|
|
|
### OpenGL Import
|
|
|
|
When using OpenGL ES, the plugin uses EGL image extension:
|
|
|
|
1. Create EGLImage from AHardwareBuffer via `eglCreateImageKHR`
|
|
2. Bind to OpenGL texture via `glEGLImageTargetTexture2DOES`
|
|
3. Blit to local texture each frame
|
|
|
|
## Device Testing
|
|
|
|
```bash
|
|
# Install MosisService first
|
|
adb install -r MosisService-debug.apk
|
|
|
|
# Install Unity app
|
|
adb install -r MosisVR.apk
|
|
|
|
# Launch service
|
|
adb shell am start -n com.omixlab.mosis/.MainActivity
|
|
|
|
# Launch Unity app
|
|
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity
|
|
|
|
# Monitor logs
|
|
adb logcat -s Unity MosisSDK Vulkan
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Black Screen / No Texture
|
|
|
|
- Verify MosisService is running (`adb shell ps | grep mosis`)
|
|
- Check logcat for connection errors
|
|
- Ensure both apps have same user ID or are debuggable
|
|
|
|
### Vulkan Errors
|
|
|
|
- Device must support `VK_ANDROID_external_memory_android_hardware_buffer`
|
|
- Check `adb logcat -s Vulkan` for extension availability
|
|
- Falls back to OpenGL if Vulkan init fails
|
|
|
|
### Touch Not Working
|
|
|
|
- Verify `PhoneInteraction` references are set
|
|
- Check XR input device selection matches controller
|
|
- Test with mouse click fallback in editor
|
|
|
|
## Implementation Notes
|
|
|
|
### Files Modified in Vulkan Implementation
|
|
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `my_native_code.cpp` | Backend abstraction, `UnityPluginLoad` detection |
|
|
| `CMakeLists.txt` | Added Vulkan library, new source files |
|
|
| `KotlinBridge.cs` | Added `SendTouchDown`/`SendTouchUp`, dynamic resolution |
|
|
| `PhoneInteraction.cs` | Added VR trigger-based touch handling |
|
|
|
|
### New Files Added
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `texture_backend.h` | ITextureBackend abstract interface |
|
|
| `opengl_backend.h/cpp` | OpenGL ES implementation |
|
|
| `vulkan_backend.h/cpp` | Vulkan implementation |
|
|
|
|
## Version History
|
|
|
|
- **1.0.0** - Initial release with OpenGL support
|
|
- **1.1.0** - Added Vulkan backend, touch down/up events, dynamic resolution
|