progress
This commit is contained in:
@@ -1 +1,221 @@
|
||||
Use this file to describe your package's features.
|
||||
# MosisSDK for Unity
|
||||
|
||||
Unity package providing integration with MosisService for virtual smartphone display in VR applications.
|
||||
|
||||
## Overview
|
||||
|
||||
This package connects to the MosisService Android app via AIDL Binder IPC to:
|
||||
- Receive rendered phone screen frames via AHardwareBuffer
|
||||
- Forward VR controller touch input to the virtual phone
|
||||
- Support both OpenGL ES and Vulkan graphics backends
|
||||
|
||||
## Requirements
|
||||
|
||||
- Unity 6000.3.2f1 or later
|
||||
- Android target platform
|
||||
- MosisService app installed on device
|
||||
|
||||
## Installation
|
||||
|
||||
1. Open Window > Package Manager
|
||||
2. Click + > Add package from disk
|
||||
3. Navigate to `Packages/com.omarator.mosissdk/package.json`
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
Packages/com.omarator.mosissdk/
|
||||
├── package.json
|
||||
├── README.md
|
||||
├── Runtime/
|
||||
│ ├── KotlinBridge.cs # C# to native bridge
|
||||
│ └── com.omarator.mosissdk.asmdef
|
||||
├── Plugins/
|
||||
│ └── Android/
|
||||
│ ├── MosisSDK.aar # Kotlin service connection
|
||||
│ ├── libs/
|
||||
│ │ └── arm64-v8a/
|
||||
│ │ └── libmy_native_lib.so
|
||||
│ └── cpp/ # Native source
|
||||
│ ├── CMakeLists.txt
|
||||
│ ├── my_native_code.cpp
|
||||
│ ├── texture_backend.h
|
||||
│ ├── opengl_backend.h/cpp
|
||||
│ └── vulkan_backend.h/cpp
|
||||
└── Assets/
|
||||
└── PhoneInteraction.cs # VR controller input
|
||||
```
|
||||
|
||||
## Build Commands
|
||||
|
||||
### Unity Build
|
||||
|
||||
1. File > Build Settings
|
||||
2. Switch Platform to Android
|
||||
3. Player Settings > Other Settings:
|
||||
- Graphics APIs: Vulkan, OpenGLES3 (in order of preference)
|
||||
- Minimum API Level: 29
|
||||
4. Build or Build and Run
|
||||
|
||||
### Native Plugin Build
|
||||
|
||||
To rebuild the native library manually:
|
||||
|
||||
```batch
|
||||
cd Packages/com.omarator.mosissdk/Plugins/Android/cpp
|
||||
|
||||
:: Configure with Android NDK
|
||||
cmake -B build ^
|
||||
-DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
|
||||
-DANDROID_ABI=arm64-v8a ^
|
||||
-DANDROID_PLATFORM=android-29
|
||||
|
||||
:: Build
|
||||
cmake --build build
|
||||
|
||||
:: Copy output
|
||||
copy build\libmy_native_lib.so ..\libs\arm64-v8a\
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Setup
|
||||
|
||||
1. Add `KotlinBridge` component to a GameObject in your scene
|
||||
2. Add `PhoneInteraction` component for VR input handling
|
||||
3. Assign VR controller and phone plane references
|
||||
|
||||
### KotlinBridge
|
||||
|
||||
The `KotlinBridge` component manages the connection to MosisService:
|
||||
|
||||
```csharp
|
||||
public class KotlinBridge : MonoBehaviour
|
||||
{
|
||||
public Material phoneMaterial; // Material to apply phone texture
|
||||
|
||||
// Called automatically when texture is ready
|
||||
// Texture is applied to phoneMaterial's _MainTex
|
||||
}
|
||||
```
|
||||
|
||||
### PhoneInteraction
|
||||
|
||||
Handles VR controller raycasting and touch input:
|
||||
|
||||
```csharp
|
||||
public class PhoneInteraction : MonoBehaviour
|
||||
{
|
||||
public GameObject Controller; // VR controller
|
||||
public GameObject Plane; // Phone display plane
|
||||
public XRNode inputDevice; // Controller hand
|
||||
public float triggerThreshold; // Trigger press threshold
|
||||
}
|
||||
```
|
||||
|
||||
### Touch API
|
||||
|
||||
Send touch events from custom scripts:
|
||||
|
||||
```csharp
|
||||
// Touch down at normalized UV (0-1)
|
||||
KotlinBridge.SendTouchDown(0.5f, 0.5f);
|
||||
|
||||
// Touch move
|
||||
KotlinBridge.SendTouchMove(0.6f, 0.5f);
|
||||
|
||||
// Touch up
|
||||
KotlinBridge.SendTouchUp(0.6f, 0.5f);
|
||||
```
|
||||
|
||||
## Graphics Backend
|
||||
|
||||
The native plugin automatically selects the appropriate backend:
|
||||
|
||||
| Graphics API | Backend | Status |
|
||||
|--------------|---------|--------|
|
||||
| Vulkan | VulkanTextureBackend | Preferred |
|
||||
| OpenGL ES 3.0 | OpenGLTextureBackend | Fallback |
|
||||
| OpenGL ES 2.0 | OpenGLTextureBackend | Fallback |
|
||||
|
||||
Backend selection happens in `UnityPluginLoad()` based on the active renderer.
|
||||
|
||||
### Vulkan Import
|
||||
|
||||
When using Vulkan, the plugin imports AHardwareBuffer directly as a VkImage:
|
||||
|
||||
1. Query buffer properties via `vkGetAndroidHardwareBufferPropertiesANDROID`
|
||||
2. Create VkImage with external memory handle type
|
||||
3. Import memory with `VkImportAndroidHardwareBufferInfoANDROID`
|
||||
4. Copy to local image each frame for safe rendering
|
||||
|
||||
### OpenGL Import
|
||||
|
||||
When using OpenGL ES, the plugin uses EGL image extension:
|
||||
|
||||
1. Create EGLImage from AHardwareBuffer via `eglCreateImageKHR`
|
||||
2. Bind to OpenGL texture via `glEGLImageTargetTexture2DOES`
|
||||
3. Blit to local texture each frame
|
||||
|
||||
## Device Testing
|
||||
|
||||
```bash
|
||||
# Install MosisService first
|
||||
adb install -r MosisService-debug.apk
|
||||
|
||||
# Install Unity app
|
||||
adb install -r MosisVR.apk
|
||||
|
||||
# Launch service
|
||||
adb shell am start -n com.omixlab.mosis/.MainActivity
|
||||
|
||||
# Launch Unity app
|
||||
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity
|
||||
|
||||
# Monitor logs
|
||||
adb logcat -s Unity MosisSDK Vulkan
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Black Screen / No Texture
|
||||
|
||||
- Verify MosisService is running (`adb shell ps | grep mosis`)
|
||||
- Check logcat for connection errors
|
||||
- Ensure both apps have same user ID or are debuggable
|
||||
|
||||
### Vulkan Errors
|
||||
|
||||
- Device must support `VK_ANDROID_external_memory_android_hardware_buffer`
|
||||
- Check `adb logcat -s Vulkan` for extension availability
|
||||
- Falls back to OpenGL if Vulkan init fails
|
||||
|
||||
### Touch Not Working
|
||||
|
||||
- Verify `PhoneInteraction` references are set
|
||||
- Check XR input device selection matches controller
|
||||
- Test with mouse click fallback in editor
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Files Modified in Vulkan Implementation
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `my_native_code.cpp` | Backend abstraction, `UnityPluginLoad` detection |
|
||||
| `CMakeLists.txt` | Added Vulkan library, new source files |
|
||||
| `KotlinBridge.cs` | Added `SendTouchDown`/`SendTouchUp`, dynamic resolution |
|
||||
| `PhoneInteraction.cs` | Added VR trigger-based touch handling |
|
||||
|
||||
### New Files Added
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `texture_backend.h` | ITextureBackend abstract interface |
|
||||
| `opengl_backend.h/cpp` | OpenGL ES implementation |
|
||||
| `vulkan_backend.h/cpp` | Vulkan implementation |
|
||||
|
||||
## Version History
|
||||
|
||||
- **1.0.0** - Initial release with OpenGL support
|
||||
- **1.1.0** - Added Vulkan backend, touch down/up events, dynamic resolution
|
||||
|
||||
Reference in New Issue
Block a user