MosisUnreal/Plugins/MosisSDK/README.md

MosisSDK Plugin for Unreal Engine 5.5

This plugin provides integration between Unreal Engine and MosisService, enabling a virtual smartphone display within VR/AR applications.

Overview

The MosisSDK plugin connects to the MosisService Android application via AIDL (Android Interface Definition Language) to:

• Receive rendered phone screen frames via AHardwareBuffer
• Forward touch input from VR controllers to the virtual phone
• Import hardware buffers into Vulkan for GPU-efficient rendering

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                         Unreal Engine Game                               │
│                                                                          │
│  ┌─────────────────┐      ┌──────────────────────────────────────────┐  │
│  │ AMosisPhoneActor│      │    UMosisPhoneTexture                    │  │
│  │  (Plane Mesh)   │      │    (extends UTexture2DDynamic)           │  │
│  │  (Material)     │      │              │                           │  │
│  └────────┬────────┘      │              ▼ CPU lock/copy             │  │
│           │               │    AHardwareBuffer_lock()                │  │
│           ▼               │              │                           │  │
│  ┌─────────────────────┐  │              ▼ RHIUpdateTexture2D        │  │
│  │ UMosisPhoneComponent│──┼──►  FTexture2DDynamicResource            │  │
│  │   (Touch Input)     │  │              │                           │  │
│  │   (Callbacks)       │  │              ▼                           │  │
│  └──────────┬──────────┘  │    UMaterialInstanceDynamic              │  │
│             │             └──────────────────────────────────────────┘  │
│             ▼                                                            │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │                          MosisClient                              │   │
│  │                   (AIDL IMosisListener)                           │   │
│  └──────────────────────────────┬───────────────────────────────────┘   │
└─────────────────────────────────┼───────────────────────────────────────┘
                                  │ Binder IPC
┌─────────────────────────────────▼───────────────────────────────────────┐
│                           MosisService                                   │
│                   (Renders phone UI via RmlUi)                           │
└─────────────────────────────────────────────────────────────────────────┘

Prerequisites

Environment Variables

Variable	Description	Example
ANDROID_HOME	Android SDK path	C:\Users\<user>\AppData\Local\Android\Sdk
ANDROID_NDK_HOME	Android NDK path (UE5 uses 25.1.8937393)	%ANDROID_HOME%\ndk\25.1.8937393

Required SDK Components

• Android SDK Platform 36 (for AIDL binder headers)
• Android Build Tools 36.1.0 (for AIDL compiler)
• Android NDK 25.1.8937393 (bundled with UE5.5)

Install via Android Studio SDK Manager:

sdkmanager "platforms;android-36" "build-tools;36.1.0"

Project Structure

Plugins/MosisSDK/
├── Source/MosisSDK/
│   ├── MosisSDK.Build.cs           # Build configuration
│   ├── MosisSDK_UPL.xml            # Android packaging rules
│   ├── AIDL/                       # AIDL interface definitions
│   │   └── com/omixlab/mosis/
│   │       ├── IMosisService.aidl
│   │       └── IMosisListener.aidl
│   ├── Generated/                  # Auto-generated headers
│   │   ├── android/
│   │   │   └── hardware_buffer_aidl.h  # Local NDK header copy
│   │   └── aidl/com/omixlab/mosis/
│   │       ├── IMosisService.h
│   │       ├── IMosisListener.h
│   │       ├── BnMosisListener.h
│   │       └── BpMosisService.h
│   ├── Public/
│   │   ├── MosisSDK.h              # Module interface
│   │   ├── MosisPhoneComponent.h   # UE5 component (touch, callbacks)
│   │   └── MosisPhoneActor.h       # Blueprint actor (mesh, material)
│   └── Private/
│       ├── MosisSDK.cpp            # Module + JNI callbacks
│       ├── MosisClient.h/cpp       # AIDL client implementation
│       ├── MosisPhoneTexture.h/cpp # UTexture2DDynamic subclass for phone screen
│       ├── MosisPhoneComponent.cpp
│       ├── MosisPhoneActor.cpp
│       └── Android/
│           └── Generated/          # AIDL-generated .cpp (Android only)
│               └── com/omixlab/mosis/
│                   ├── IMosisService.cpp
│                   └── IMosisListener.cpp
└── Binaries/
    └── Win64/                      # Windows editor binaries
    └── Android/                    # Android binaries

Build Instructions

Windows Editor Build

Build the plugin for use in the Unreal Editor:

"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
    MosisUnrealEditor Win64 Development ^
    -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"

Output: Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll

Android Build

Build and package for Android deployment:

"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
    BuildCookRun ^
    -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
    -platform=Android ^
    -clientconfig=Development ^
    -build -cook -stage -pak -package ^
    -noP4 -utf8output

Output: Binaries/Android/MosisUnreal-arm64.apk

Clean Build

To force a full rebuild:

rmdir /s /q "Intermediate\Build"
rmdir /s /q "Binaries"
rmdir /s /q "Plugins\MosisSDK\Intermediate"
rmdir /s /q "Plugins\MosisSDK\Binaries"

Key Implementation Details

AIDL Code Generation

The Build.cs automatically runs the AIDL compiler during Android builds:

1. Input: .aidl files in Source/MosisSDK/AIDL/
2. Header Output: Source/MosisSDK/Generated/ (included for all platforms)
3. CPP Output: Source/MosisSDK/Private/Android/Generated/ (Android only)

The separation ensures Windows builds don't try to compile Android-specific AIDL bindings.

Platform-Specific Dependencies

Dependency	Windows	Android	Purpose
Core, CoreUObject, Engine	✓	✓	UE5 fundamentals
Slate, SlateCore	✓	✓	UI framework
RenderCore, RHI	✓	✓	Rendering abstraction
Launch, ApplicationCore	✗	✓	Android JNI access
VulkanRHI	✗	✓	IVulkanDynamicRHI for HardwareBuffer import
Vulkan	✗	✓	Vulkan headers
binder_ndk, android, nativewindow	✗	✓	Android system libs

NDK Header Compatibility

UE5.5 uses NDK 25, but hardware_buffer_aidl.h was added in NDK 26+. The plugin includes a local copy with modifications:

1. Removed __INTRODUCED_IN(34) annotations (causes API level errors)
2. Added __attribute__((weak)) to parcel functions (allows linking on API 33)

MosisClient (IMosisListener Implementation)

The MosisClient class implements the AIDL listener interface:

class MosisClient : public aidl::com::omixlab::mosis::BnMosisListener
{
public:
    // Called when service initializes
    ndk::ScopedAStatus onServiceInitialized(bool success) override;

    // Called when a new hardware buffer is available
    ndk::ScopedAStatus onBufferAvailable(const HardwareBuffer& buffer) override;

    // Called each frame when content updates
    ndk::ScopedAStatus onFrameAvailable() override;

    // Touch forwarding to service
    void SendTouchDown(float x, float y);
    void SendTouchMove(float x, float y);
    void SendTouchUp(float x, float y);
};

Service Connection Flow

1. FMosisSDKModule::StartupModule() calls Java MyKotlinPlugin.StartMosisService()
2. Kotlin code binds to MosisService
3. On connection, Kotlin calls native serviceConnected(IBinder binder)
4. JNI callback creates MosisClient from the binder
5. Client calls initOS() on the service
6. Service sends onBufferAvailable() with the shared hardware buffer
7. Client imports buffer into Vulkan texture
8. Service sends onFrameAvailable() each rendered frame

Usage in Blueprints

MosisPhoneActor

Drop AMosisPhoneActor into your level for a ready-to-use phone display:

1. Add to level via Place Actors > MosisPhoneActor
2. Configure material and mesh as needed
3. The actor automatically connects to MosisService on BeginPlay

MosisPhoneComponent

For custom actors, add UMosisPhoneComponent:

UPROPERTY(VisibleAnywhere)
UMosisPhoneComponent* PhoneComponent;

// In constructor
PhoneComponent = CreateDefaultSubobject<UMosisPhoneComponent>(TEXT("Phone"));

Touch Input

Send touch events from VR controller raycasts:

// In your VR interaction component
FVector2D UV = CalculateHitUV(HitResult);

if (TriggerPressed && !WasTriggerPressed)
    PhoneComponent->SendTouch(UV, EMosisTouchType::Down);
else if (TriggerPressed)
    PhoneComponent->SendTouch(UV, EMosisTouchType::Move);
else if (!TriggerPressed && WasTriggerPressed)
    PhoneComponent->SendTouch(UV, EMosisTouchType::Up);

Troubleshooting

Build Errors

"Cannot open include file: 'aidl/com/omixlab/mosis/IMosisListener.h'"

• Ensure Android SDK Platform 36 is installed
• Check ANDROID_HOME environment variable
• Run Android build first to generate headers

"LINK : fatal error LNK1181: cannot open input file 'UnrealEditor.lib'"

• The Launch module was incorrectly in global dependencies
• Should only be in Android-specific dependencies
• Solution: Move Launch to the Android platform block in Build.cs

"'Android/AndroidJNI.h' file not found"

• Add Launch and ApplicationCore to Android dependencies in Build.cs

"Cannot open include file: 'android/hardware_buffer_aidl.h'"

• This header is only in NDK 26+, but UE5.5 uses NDK 25
• Solution: Local copy in Generated/android/hardware_buffer_aidl.h

"'AHardwareBuffer_readFromParcel' is unavailable" (API level errors)

• Functions marked as API 34+ but building for API 33
• Solution: Add __attribute__((weak)) to function declarations in local header

AIDL cpp files compiled on Windows causing errors

• AIDL-generated .cpp files contain Android-only headers
• Solution: Output cpp to Private/Android/Generated/ (only compiled on Android)
• Headers go to Generated/ (included but not compiled)

Runtime Issues

Service not connecting

• Ensure MosisService app is installed and running
• Check logcat: adb logcat -s MosisSDK MosisTest
• Verify package name matches in MyKotlinPlugin.kt

Black texture / No frames

• Check onBufferAvailable callback is received
• Verify Vulkan extensions are supported on device
• Check hardware buffer format compatibility

Device Testing

Installation

The Android build produces both an APK and an OBB file. Both must be deployed for each build:

:: Install APK
adb install -r Binaries\Android\MosisUnreal-arm64.apk

:: Create OBB directory and push OBB file
adb shell mkdir -p /sdcard/Android/obb/com.omixlab.MosisUnreal
adb push Binaries\Android\main.1.com.omixlab.MosisUnreal.obb /sdcard/Android/obb/com.omixlab.MosisUnreal/

:: Install MosisService
adb install -r path\to\MosisService.apk

Alternatively, use the generated install script:

cd Binaries\Android
Install_MosisUnreal-arm64.bat

Launching

:: Launch service first
adb shell am start -n com.omixlab.mosis/.MainActivity

:: Launch game
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity

:: Monitor logs
adb logcat -s MosisSDK MosisTest RMLUI LogMosisClient LogMosisSDK

Expected Log Output

Successful connection shows:

LogMosisSDK: serviceConnected callback received
LogMosisClient: onServiceInitialized: true
LogMosisClient: Create: initOS returned true
LogMosisSDK: serviceConnected: MosisClient created successfully
LogMosisClient: onBufferAvailable: 540x960, format=1
LogMosisSDK: Buffer callback: buffer=0x...

Meta Quest Configuration

Hand Tracking Support

The plugin includes hand tracking support to allow launching on Quest devices without controllers. This is configured in MosisSDK_UPL.xml:

<uses-feature android:name="oculus.software.handtracking" android:required="false" />
<uses-permission android:name="horizonos.permission.HAND_TRACKING" />

Note: Uses horizonos.permission.HAND_TRACKING (not the deprecated com.oculus.permission.HAND_TRACKING).

OculusXR Settings

Hand tracking is also enabled in DefaultEngine.ini:

[/Script/OculusXRHMD.OculusXRHMDRuntimeSettings]
bSupportHandTracking=True
HandTrackingSupport=ControllersAndHands
HandTrackingFrequency=High
HandTrackingVersion=V2

Version History

• v1.2 - UTexture2DDynamic integration

  • Switched UMosisPhoneTexture to extend UTexture2DDynamic for proper material system integration
  • Removed custom FMosisPhoneTextureResource - using UE5's built-in FTexture2DDynamicResource
  • CPU-based texture upload via AHardwareBuffer_lock() and RHIUpdateTexture2D()
  • Fixed stride conversion (AHardwareBuffer stride is in pixels, RHI expects bytes)

• v1.1 - UE5 RHI texture integration

  • New UMosisPhoneTexture using UE5's IVulkanDynamicRHI::RHICreateTexture2DFromAndroidHardwareBuffer()
  • FMosisPhoneTextureResource for render thread HardwareBuffer import
  • AMosisPhoneActor now includes default plane mesh and material setup
  • Thread-safe callbacks using TWeakObjectPtr and TAtomic
  • Automatic texture-to-material binding in tick

• v1.0 - Initial implementation

  • AIDL client for MosisService connection
  • Vulkan HardwareBuffer import
  • UE5 component and actor for phone display
  • Touch input forwarding