380 lines
15 KiB
Markdown
380 lines
15 KiB
Markdown
# MosisSDK Plugin for Unreal Engine 5.5
|
|
|
|
This plugin provides integration between Unreal Engine and MosisService, enabling a virtual smartphone display within VR/AR applications.
|
|
|
|
## Overview
|
|
|
|
The MosisSDK plugin connects to the MosisService Android application via AIDL (Android Interface Definition Language) to:
|
|
- Receive rendered phone screen frames via AHardwareBuffer
|
|
- Forward touch input from VR controllers to the virtual phone
|
|
- Import hardware buffers into Vulkan for GPU-efficient rendering
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ Unreal Engine Game │
|
|
│ │
|
|
│ ┌─────────────────┐ ┌──────────────────────────────────────────┐ │
|
|
│ │ AMosisPhoneActor│ │ UMosisPhoneTexture │ │
|
|
│ │ (Plane Mesh) │ │ │ │ │
|
|
│ │ (Material) │ │ ▼ │ │
|
|
│ └────────┬────────┘ │ ┌────────────────────────────────────┐ │ │
|
|
│ │ │ │ FMosisPhoneTextureResource │ │ │
|
|
│ ▼ │ │ │ │ │ │
|
|
│ ┌─────────────────────┐ │ │ ▼ ENQUEUE_RENDER_CMD │ │ │
|
|
│ │ UMosisPhoneComponent│──┼──► IVulkanDynamicRHI:: │ │ │
|
|
│ │ (Touch Input) │ │ │ RHICreateTexture2DFromAndroid │ │ │
|
|
│ │ (Callbacks) │ │ │ HardwareBuffer() │ │ │
|
|
│ └──────────┬──────────┘ │ └────────────────────────────────────┘ │ │
|
|
│ │ └──────────────────────────────────────────┘ │
|
|
│ ▼ │
|
|
│ ┌──────────────────────────────────────────────────────────────────┐ │
|
|
│ │ MosisClient │ │
|
|
│ │ (AIDL IMosisListener) │ │
|
|
│ └──────────────────────────────┬───────────────────────────────────┘ │
|
|
└─────────────────────────────────┼───────────────────────────────────────┘
|
|
│ Binder IPC
|
|
┌─────────────────────────────────▼───────────────────────────────────────┐
|
|
│ MosisService │
|
|
│ (Renders phone UI via RmlUi) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Description | Example |
|
|
|----------|-------------|---------|
|
|
| `ANDROID_HOME` | Android SDK path | `C:\Users\<user>\AppData\Local\Android\Sdk` |
|
|
| `ANDROID_NDK_HOME` | Android NDK path (UE5 uses 25.1.8937393) | `%ANDROID_HOME%\ndk\25.1.8937393` |
|
|
|
|
### Required SDK Components
|
|
|
|
- Android SDK Platform 36 (for AIDL binder headers)
|
|
- Android Build Tools 36.1.0 (for AIDL compiler)
|
|
- Android NDK 25.1.8937393 (bundled with UE5.5)
|
|
|
|
Install via Android Studio SDK Manager:
|
|
```
|
|
sdkmanager "platforms;android-36" "build-tools;36.1.0"
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
Plugins/MosisSDK/
|
|
├── Source/MosisSDK/
|
|
│ ├── MosisSDK.Build.cs # Build configuration
|
|
│ ├── MosisSDK_UPL.xml # Android packaging rules
|
|
│ ├── AIDL/ # AIDL interface definitions
|
|
│ │ └── com/omixlab/mosis/
|
|
│ │ ├── IMosisService.aidl
|
|
│ │ └── IMosisListener.aidl
|
|
│ ├── Generated/ # Auto-generated headers
|
|
│ │ ├── android/
|
|
│ │ │ └── hardware_buffer_aidl.h # Local NDK header copy
|
|
│ │ └── aidl/com/omixlab/mosis/
|
|
│ │ ├── IMosisService.h
|
|
│ │ ├── IMosisListener.h
|
|
│ │ ├── BnMosisListener.h
|
|
│ │ └── BpMosisService.h
|
|
│ ├── Public/
|
|
│ │ ├── MosisSDK.h # Module interface
|
|
│ │ ├── MosisPhoneComponent.h # UE5 component (touch, callbacks)
|
|
│ │ └── MosisPhoneActor.h # Blueprint actor (mesh, material)
|
|
│ └── Private/
|
|
│ ├── MosisSDK.cpp # Module + JNI callbacks
|
|
│ ├── MosisClient.h/cpp # AIDL client implementation
|
|
│ ├── MosisPhoneTexture.h/cpp # UTexture for phone screen
|
|
│ ├── MosisPhoneTextureResource.h/cpp # Render thread resource
|
|
│ ├── MosisVulkanTexture.h/cpp # Legacy Vulkan import (unused)
|
|
│ ├── MosisPhoneComponent.cpp
|
|
│ ├── MosisPhoneActor.cpp
|
|
│ └── Android/
|
|
│ └── Generated/ # AIDL-generated .cpp (Android only)
|
|
│ └── com/omixlab/mosis/
|
|
│ ├── IMosisService.cpp
|
|
│ └── IMosisListener.cpp
|
|
└── Binaries/
|
|
└── Win64/ # Windows editor binaries
|
|
└── Android/ # Android binaries
|
|
```
|
|
|
|
## Build Instructions
|
|
|
|
### Windows Editor Build
|
|
|
|
Build the plugin for use in the Unreal Editor:
|
|
|
|
```batch
|
|
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
|
|
MosisUnrealEditor Win64 Development ^
|
|
-Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"
|
|
```
|
|
|
|
**Output**: `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll`
|
|
|
|
### Android Build
|
|
|
|
Build and package for Android deployment:
|
|
|
|
```batch
|
|
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
|
|
BuildCookRun ^
|
|
-project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
|
|
-platform=Android ^
|
|
-clientconfig=Development ^
|
|
-build -cook -stage -pak -package ^
|
|
-noP4 -utf8output
|
|
```
|
|
|
|
**Output**: `Binaries/Android/MosisUnreal-arm64.apk`
|
|
|
|
### Clean Build
|
|
|
|
To force a full rebuild:
|
|
|
|
```batch
|
|
rmdir /s /q "Intermediate\Build"
|
|
rmdir /s /q "Binaries"
|
|
rmdir /s /q "Plugins\MosisSDK\Intermediate"
|
|
rmdir /s /q "Plugins\MosisSDK\Binaries"
|
|
```
|
|
|
|
## Key Implementation Details
|
|
|
|
### AIDL Code Generation
|
|
|
|
The Build.cs automatically runs the AIDL compiler during Android builds:
|
|
|
|
1. **Input**: `.aidl` files in `Source/MosisSDK/AIDL/`
|
|
2. **Header Output**: `Source/MosisSDK/Generated/` (included for all platforms)
|
|
3. **CPP Output**: `Source/MosisSDK/Private/Android/Generated/` (Android only)
|
|
|
|
The separation ensures Windows builds don't try to compile Android-specific AIDL bindings.
|
|
|
|
### Platform-Specific Dependencies
|
|
|
|
| Dependency | Windows | Android | Purpose |
|
|
|------------|---------|---------|---------|
|
|
| Core, CoreUObject, Engine | ✓ | ✓ | UE5 fundamentals |
|
|
| Slate, SlateCore | ✓ | ✓ | UI framework |
|
|
| RenderCore, RHI | ✓ | ✓ | Rendering abstraction |
|
|
| Launch, ApplicationCore | ✗ | ✓ | Android JNI access |
|
|
| VulkanRHI | ✗ | ✓ | IVulkanDynamicRHI for HardwareBuffer import |
|
|
| Vulkan | ✗ | ✓ | Vulkan headers |
|
|
| binder_ndk, android, nativewindow | ✗ | ✓ | Android system libs |
|
|
|
|
### NDK Header Compatibility
|
|
|
|
UE5.5 uses NDK 25, but `hardware_buffer_aidl.h` was added in NDK 26+. The plugin includes a local copy with modifications:
|
|
|
|
1. Removed `__INTRODUCED_IN(34)` annotations (causes API level errors)
|
|
2. Added `__attribute__((weak))` to parcel functions (allows linking on API 33)
|
|
|
|
### MosisClient (IMosisListener Implementation)
|
|
|
|
The `MosisClient` class implements the AIDL listener interface:
|
|
|
|
```cpp
|
|
class MosisClient : public aidl::com::omixlab::mosis::BnMosisListener
|
|
{
|
|
public:
|
|
// Called when service initializes
|
|
ndk::ScopedAStatus onServiceInitialized(bool success) override;
|
|
|
|
// Called when a new hardware buffer is available
|
|
ndk::ScopedAStatus onBufferAvailable(const HardwareBuffer& buffer) override;
|
|
|
|
// Called each frame when content updates
|
|
ndk::ScopedAStatus onFrameAvailable() override;
|
|
|
|
// Touch forwarding to service
|
|
void SendTouchDown(float x, float y);
|
|
void SendTouchMove(float x, float y);
|
|
void SendTouchUp(float x, float y);
|
|
};
|
|
```
|
|
|
|
### Service Connection Flow
|
|
|
|
1. `FMosisSDKModule::StartupModule()` calls Java `MyKotlinPlugin.StartMosisService()`
|
|
2. Kotlin code binds to MosisService
|
|
3. On connection, Kotlin calls native `serviceConnected(IBinder binder)`
|
|
4. JNI callback creates `MosisClient` from the binder
|
|
5. Client calls `initOS()` on the service
|
|
6. Service sends `onBufferAvailable()` with the shared hardware buffer
|
|
7. Client imports buffer into Vulkan texture
|
|
8. Service sends `onFrameAvailable()` each rendered frame
|
|
|
|
## Usage in Blueprints
|
|
|
|
### MosisPhoneActor
|
|
|
|
Drop `AMosisPhoneActor` into your level for a ready-to-use phone display:
|
|
|
|
1. Add to level via Place Actors > MosisPhoneActor
|
|
2. Configure material and mesh as needed
|
|
3. The actor automatically connects to MosisService on BeginPlay
|
|
|
|
### MosisPhoneComponent
|
|
|
|
For custom actors, add `UMosisPhoneComponent`:
|
|
|
|
```cpp
|
|
UPROPERTY(VisibleAnywhere)
|
|
UMosisPhoneComponent* PhoneComponent;
|
|
|
|
// In constructor
|
|
PhoneComponent = CreateDefaultSubobject<UMosisPhoneComponent>(TEXT("Phone"));
|
|
```
|
|
|
|
### Touch Input
|
|
|
|
Send touch events from VR controller raycasts:
|
|
|
|
```cpp
|
|
// In your VR interaction component
|
|
FVector2D UV = CalculateHitUV(HitResult);
|
|
|
|
if (TriggerPressed && !WasTriggerPressed)
|
|
PhoneComponent->SendTouch(UV, EMosisTouchType::Down);
|
|
else if (TriggerPressed)
|
|
PhoneComponent->SendTouch(UV, EMosisTouchType::Move);
|
|
else if (!TriggerPressed && WasTriggerPressed)
|
|
PhoneComponent->SendTouch(UV, EMosisTouchType::Up);
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Build Errors
|
|
|
|
**"Cannot open include file: 'aidl/com/omixlab/mosis/IMosisListener.h'"**
|
|
- Ensure Android SDK Platform 36 is installed
|
|
- Check `ANDROID_HOME` environment variable
|
|
- Run Android build first to generate headers
|
|
|
|
**"LINK : fatal error LNK1181: cannot open input file 'UnrealEditor.lib'"**
|
|
- The `Launch` module was incorrectly in global dependencies
|
|
- Should only be in Android-specific dependencies
|
|
- Solution: Move `Launch` to the Android platform block in Build.cs
|
|
|
|
**"'Android/AndroidJNI.h' file not found"**
|
|
- Add `Launch` and `ApplicationCore` to Android dependencies in Build.cs
|
|
|
|
**"Cannot open include file: 'android/hardware_buffer_aidl.h'"**
|
|
- This header is only in NDK 26+, but UE5.5 uses NDK 25
|
|
- Solution: Local copy in `Generated/android/hardware_buffer_aidl.h`
|
|
|
|
**"'AHardwareBuffer_readFromParcel' is unavailable" (API level errors)**
|
|
- Functions marked as API 34+ but building for API 33
|
|
- Solution: Add `__attribute__((weak))` to function declarations in local header
|
|
|
|
**AIDL cpp files compiled on Windows causing errors**
|
|
- AIDL-generated .cpp files contain Android-only headers
|
|
- Solution: Output cpp to `Private/Android/Generated/` (only compiled on Android)
|
|
- Headers go to `Generated/` (included but not compiled)
|
|
|
|
### Runtime Issues
|
|
|
|
**Service not connecting**
|
|
- Ensure MosisService app is installed and running
|
|
- Check logcat: `adb logcat -s MosisSDK MosisTest`
|
|
- Verify package name matches in `MyKotlinPlugin.kt`
|
|
|
|
**Black texture / No frames**
|
|
- Check `onBufferAvailable` callback is received
|
|
- Verify Vulkan extensions are supported on device
|
|
- Check hardware buffer format compatibility
|
|
|
|
## Device Testing
|
|
|
|
### Installation
|
|
|
|
The Android build produces both an APK and an OBB file. **Both must be deployed for each build**:
|
|
|
|
```batch
|
|
:: Install APK
|
|
adb install -r Binaries\Android\MosisUnreal-arm64.apk
|
|
|
|
:: Create OBB directory and push OBB file
|
|
adb shell mkdir -p /sdcard/Android/obb/com.omixlab.MosisUnreal
|
|
adb push Binaries\Android\main.1.com.omixlab.MosisUnreal.obb /sdcard/Android/obb/com.omixlab.MosisUnreal/
|
|
|
|
:: Install MosisService
|
|
adb install -r path\to\MosisService.apk
|
|
```
|
|
|
|
Alternatively, use the generated install script:
|
|
```batch
|
|
cd Binaries\Android
|
|
Install_MosisUnreal-arm64.bat
|
|
```
|
|
|
|
### Launching
|
|
|
|
```batch
|
|
:: Launch service first
|
|
adb shell am start -n com.omixlab.mosis/.MainActivity
|
|
|
|
:: Launch game
|
|
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
|
|
|
:: Monitor logs
|
|
adb logcat -s MosisSDK MosisTest RMLUI LogMosisClient LogMosisSDK
|
|
```
|
|
|
|
### Expected Log Output
|
|
|
|
Successful connection shows:
|
|
```
|
|
LogMosisSDK: serviceConnected callback received
|
|
LogMosisClient: onServiceInitialized: true
|
|
LogMosisClient: Create: initOS returned true
|
|
LogMosisSDK: serviceConnected: MosisClient created successfully
|
|
LogMosisClient: onBufferAvailable: 540x960, format=1
|
|
LogMosisSDK: Buffer callback: buffer=0x...
|
|
```
|
|
|
|
## Meta Quest Configuration
|
|
|
|
### Hand Tracking Support
|
|
|
|
The plugin includes hand tracking support to allow launching on Quest devices without controllers. This is configured in `MosisSDK_UPL.xml`:
|
|
|
|
```xml
|
|
<uses-feature android:name="oculus.software.handtracking" android:required="false" />
|
|
<uses-permission android:name="horizonos.permission.HAND_TRACKING" />
|
|
```
|
|
|
|
**Note**: Uses `horizonos.permission.HAND_TRACKING` (not the deprecated `com.oculus.permission.HAND_TRACKING`).
|
|
|
|
### OculusXR Settings
|
|
|
|
Hand tracking is also enabled in `DefaultEngine.ini`:
|
|
|
|
```ini
|
|
[/Script/OculusXRHMD.OculusXRHMDRuntimeSettings]
|
|
bSupportHandTracking=True
|
|
HandTrackingSupport=ControllersAndHands
|
|
HandTrackingFrequency=High
|
|
HandTrackingVersion=V2
|
|
```
|
|
|
|
## Version History
|
|
|
|
- **v1.1** - UE5 RHI texture integration
|
|
- New `UMosisPhoneTexture` using UE5's `IVulkanDynamicRHI::RHICreateTexture2DFromAndroidHardwareBuffer()`
|
|
- `FMosisPhoneTextureResource` for render thread HardwareBuffer import
|
|
- `AMosisPhoneActor` now includes default plane mesh and material setup
|
|
- Thread-safe callbacks using `TWeakObjectPtr` and `TAtomic`
|
|
- Automatic texture-to-material binding in tick
|
|
|
|
- **v1.0** - Initial implementation
|
|
- AIDL client for MosisService connection
|
|
- Vulkan HardwareBuffer import
|
|
- UE5 component and actor for phone display
|
|
- Touch input forwarding
|