# MosisSDK Plugin for Unreal Engine 5.5 This plugin provides integration between Unreal Engine and MosisService, enabling a virtual smartphone display within VR/AR applications. ## Overview The MosisSDK plugin connects to the MosisService Android application via AIDL (Android Interface Definition Language) to: - Receive rendered phone screen frames via AHardwareBuffer - Forward touch input from VR controllers to the virtual phone - Import hardware buffers into Vulkan for GPU-efficient rendering ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Unreal Engine Game │ │ │ │ ┌─────────────────┐ ┌──────────────────────────────────────────┐ │ │ │ AMosisPhoneActor│ │ UMosisPhoneTexture │ │ │ │ (Plane Mesh) │ │ │ │ │ │ │ (Material) │ │ ▼ │ │ │ └────────┬────────┘ │ ┌────────────────────────────────────┐ │ │ │ │ │ │ FMosisPhoneTextureResource │ │ │ │ ▼ │ │ │ │ │ │ │ ┌─────────────────────┐ │ │ ▼ ENQUEUE_RENDER_CMD │ │ │ │ │ UMosisPhoneComponent│──┼──► IVulkanDynamicRHI:: │ │ │ │ │ (Touch Input) │ │ │ RHICreateTexture2DFromAndroid │ │ │ │ │ (Callbacks) │ │ │ HardwareBuffer() │ │ │ │ └──────────┬──────────┘ │ └────────────────────────────────────┘ │ │ │ │ └──────────────────────────────────────────┘ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ │ MosisClient │ │ │ │ (AIDL IMosisListener) │ │ │ └──────────────────────────────┬───────────────────────────────────┘ │ └─────────────────────────────────┼───────────────────────────────────────┘ │ Binder IPC ┌─────────────────────────────────▼───────────────────────────────────────┐ │ MosisService │ │ (Renders phone UI via RmlUi) │ └─────────────────────────────────────────────────────────────────────────┘ ``` ## Prerequisites ### Environment Variables | Variable | Description | Example | |----------|-------------|---------| | `ANDROID_HOME` | Android SDK path | `C:\Users\\AppData\Local\Android\Sdk` | | `ANDROID_NDK_HOME` | Android NDK path (UE5 uses 25.1.8937393) | `%ANDROID_HOME%\ndk\25.1.8937393` | ### Required SDK Components - Android SDK Platform 36 (for AIDL binder headers) - Android Build Tools 36.1.0 (for AIDL compiler) - Android NDK 25.1.8937393 (bundled with UE5.5) Install via Android Studio SDK Manager: ``` sdkmanager "platforms;android-36" "build-tools;36.1.0" ``` ## Project Structure ``` Plugins/MosisSDK/ ├── Source/MosisSDK/ │ ├── MosisSDK.Build.cs # Build configuration │ ├── MosisSDK_UPL.xml # Android packaging rules │ ├── AIDL/ # AIDL interface definitions │ │ └── com/omixlab/mosis/ │ │ ├── IMosisService.aidl │ │ └── IMosisListener.aidl │ ├── Generated/ # Auto-generated headers │ │ ├── android/ │ │ │ └── hardware_buffer_aidl.h # Local NDK header copy │ │ └── aidl/com/omixlab/mosis/ │ │ ├── IMosisService.h │ │ ├── IMosisListener.h │ │ ├── BnMosisListener.h │ │ └── BpMosisService.h │ ├── Public/ │ │ ├── MosisSDK.h # Module interface │ │ ├── MosisPhoneComponent.h # UE5 component (touch, callbacks) │ │ └── MosisPhoneActor.h # Blueprint actor (mesh, material) │ └── Private/ │ ├── MosisSDK.cpp # Module + JNI callbacks │ ├── MosisClient.h/cpp # AIDL client implementation │ ├── MosisPhoneTexture.h/cpp # UTexture for phone screen │ ├── MosisPhoneTextureResource.h/cpp # Render thread resource │ ├── MosisPhoneComponent.cpp │ ├── MosisPhoneActor.cpp │ └── Android/ │ └── Generated/ # AIDL-generated .cpp (Android only) │ └── com/omixlab/mosis/ │ ├── IMosisService.cpp │ └── IMosisListener.cpp └── Binaries/ └── Win64/ # Windows editor binaries └── Android/ # Android binaries ``` ## Build Instructions ### Windows Editor Build Build the plugin for use in the Unreal Editor: ```batch "D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^ MosisUnrealEditor Win64 Development ^ -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ``` **Output**: `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll` ### Android Build Build and package for Android deployment: ```batch "D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^ BuildCookRun ^ -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^ -platform=Android ^ -clientconfig=Development ^ -build -cook -stage -pak -package ^ -noP4 -utf8output ``` **Output**: `Binaries/Android/MosisUnreal-arm64.apk` ### Clean Build To force a full rebuild: ```batch rmdir /s /q "Intermediate\Build" rmdir /s /q "Binaries" rmdir /s /q "Plugins\MosisSDK\Intermediate" rmdir /s /q "Plugins\MosisSDK\Binaries" ``` ## Key Implementation Details ### AIDL Code Generation The Build.cs automatically runs the AIDL compiler during Android builds: 1. **Input**: `.aidl` files in `Source/MosisSDK/AIDL/` 2. **Header Output**: `Source/MosisSDK/Generated/` (included for all platforms) 3. **CPP Output**: `Source/MosisSDK/Private/Android/Generated/` (Android only) The separation ensures Windows builds don't try to compile Android-specific AIDL bindings. ### Platform-Specific Dependencies | Dependency | Windows | Android | Purpose | |------------|---------|---------|---------| | Core, CoreUObject, Engine | ✓ | ✓ | UE5 fundamentals | | Slate, SlateCore | ✓ | ✓ | UI framework | | RenderCore, RHI | ✓ | ✓ | Rendering abstraction | | Launch, ApplicationCore | ✗ | ✓ | Android JNI access | | VulkanRHI | ✗ | ✓ | IVulkanDynamicRHI for HardwareBuffer import | | Vulkan | ✗ | ✓ | Vulkan headers | | binder_ndk, android, nativewindow | ✗ | ✓ | Android system libs | ### NDK Header Compatibility UE5.5 uses NDK 25, but `hardware_buffer_aidl.h` was added in NDK 26+. The plugin includes a local copy with modifications: 1. Removed `__INTRODUCED_IN(34)` annotations (causes API level errors) 2. Added `__attribute__((weak))` to parcel functions (allows linking on API 33) ### MosisClient (IMosisListener Implementation) The `MosisClient` class implements the AIDL listener interface: ```cpp class MosisClient : public aidl::com::omixlab::mosis::BnMosisListener { public: // Called when service initializes ndk::ScopedAStatus onServiceInitialized(bool success) override; // Called when a new hardware buffer is available ndk::ScopedAStatus onBufferAvailable(const HardwareBuffer& buffer) override; // Called each frame when content updates ndk::ScopedAStatus onFrameAvailable() override; // Touch forwarding to service void SendTouchDown(float x, float y); void SendTouchMove(float x, float y); void SendTouchUp(float x, float y); }; ``` ### Service Connection Flow 1. `FMosisSDKModule::StartupModule()` calls Java `MyKotlinPlugin.StartMosisService()` 2. Kotlin code binds to MosisService 3. On connection, Kotlin calls native `serviceConnected(IBinder binder)` 4. JNI callback creates `MosisClient` from the binder 5. Client calls `initOS()` on the service 6. Service sends `onBufferAvailable()` with the shared hardware buffer 7. Client imports buffer into Vulkan texture 8. Service sends `onFrameAvailable()` each rendered frame ## Usage in Blueprints ### MosisPhoneActor Drop `AMosisPhoneActor` into your level for a ready-to-use phone display: 1. Add to level via Place Actors > MosisPhoneActor 2. Configure material and mesh as needed 3. The actor automatically connects to MosisService on BeginPlay ### MosisPhoneComponent For custom actors, add `UMosisPhoneComponent`: ```cpp UPROPERTY(VisibleAnywhere) UMosisPhoneComponent* PhoneComponent; // In constructor PhoneComponent = CreateDefaultSubobject(TEXT("Phone")); ``` ### Touch Input Send touch events from VR controller raycasts: ```cpp // In your VR interaction component FVector2D UV = CalculateHitUV(HitResult); if (TriggerPressed && !WasTriggerPressed) PhoneComponent->SendTouch(UV, EMosisTouchType::Down); else if (TriggerPressed) PhoneComponent->SendTouch(UV, EMosisTouchType::Move); else if (!TriggerPressed && WasTriggerPressed) PhoneComponent->SendTouch(UV, EMosisTouchType::Up); ``` ## Troubleshooting ### Build Errors **"Cannot open include file: 'aidl/com/omixlab/mosis/IMosisListener.h'"** - Ensure Android SDK Platform 36 is installed - Check `ANDROID_HOME` environment variable - Run Android build first to generate headers **"LINK : fatal error LNK1181: cannot open input file 'UnrealEditor.lib'"** - The `Launch` module was incorrectly in global dependencies - Should only be in Android-specific dependencies - Solution: Move `Launch` to the Android platform block in Build.cs **"'Android/AndroidJNI.h' file not found"** - Add `Launch` and `ApplicationCore` to Android dependencies in Build.cs **"Cannot open include file: 'android/hardware_buffer_aidl.h'"** - This header is only in NDK 26+, but UE5.5 uses NDK 25 - Solution: Local copy in `Generated/android/hardware_buffer_aidl.h` **"'AHardwareBuffer_readFromParcel' is unavailable" (API level errors)** - Functions marked as API 34+ but building for API 33 - Solution: Add `__attribute__((weak))` to function declarations in local header **AIDL cpp files compiled on Windows causing errors** - AIDL-generated .cpp files contain Android-only headers - Solution: Output cpp to `Private/Android/Generated/` (only compiled on Android) - Headers go to `Generated/` (included but not compiled) ### Runtime Issues **Service not connecting** - Ensure MosisService app is installed and running - Check logcat: `adb logcat -s MosisSDK MosisTest` - Verify package name matches in `MyKotlinPlugin.kt` **Black texture / No frames** - Check `onBufferAvailable` callback is received - Verify Vulkan extensions are supported on device - Check hardware buffer format compatibility ## Device Testing ### Installation The Android build produces both an APK and an OBB file. **Both must be deployed for each build**: ```batch :: Install APK adb install -r Binaries\Android\MosisUnreal-arm64.apk :: Create OBB directory and push OBB file adb shell mkdir -p /sdcard/Android/obb/com.omixlab.MosisUnreal adb push Binaries\Android\main.1.com.omixlab.MosisUnreal.obb /sdcard/Android/obb/com.omixlab.MosisUnreal/ :: Install MosisService adb install -r path\to\MosisService.apk ``` Alternatively, use the generated install script: ```batch cd Binaries\Android Install_MosisUnreal-arm64.bat ``` ### Launching ```batch :: Launch service first adb shell am start -n com.omixlab.mosis/.MainActivity :: Launch game adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity :: Monitor logs adb logcat -s MosisSDK MosisTest RMLUI LogMosisClient LogMosisSDK ``` ### Expected Log Output Successful connection shows: ``` LogMosisSDK: serviceConnected callback received LogMosisClient: onServiceInitialized: true LogMosisClient: Create: initOS returned true LogMosisSDK: serviceConnected: MosisClient created successfully LogMosisClient: onBufferAvailable: 540x960, format=1 LogMosisSDK: Buffer callback: buffer=0x... ``` ## Meta Quest Configuration ### Hand Tracking Support The plugin includes hand tracking support to allow launching on Quest devices without controllers. This is configured in `MosisSDK_UPL.xml`: ```xml ``` **Note**: Uses `horizonos.permission.HAND_TRACKING` (not the deprecated `com.oculus.permission.HAND_TRACKING`). ### OculusXR Settings Hand tracking is also enabled in `DefaultEngine.ini`: ```ini [/Script/OculusXRHMD.OculusXRHMDRuntimeSettings] bSupportHandTracking=True HandTrackingSupport=ControllersAndHands HandTrackingFrequency=High HandTrackingVersion=V2 ``` ## Version History - **v1.1** - UE5 RHI texture integration - New `UMosisPhoneTexture` using UE5's `IVulkanDynamicRHI::RHICreateTexture2DFromAndroidHardwareBuffer()` - `FMosisPhoneTextureResource` for render thread HardwareBuffer import - `AMosisPhoneActor` now includes default plane mesh and material setup - Thread-safe callbacks using `TWeakObjectPtr` and `TAtomic` - Automatic texture-to-material binding in tick - **v1.0** - Initial implementation - AIDL client for MosisService connection - Vulkan HardwareBuffer import - UE5 component and actor for phone display - Touch input forwarding