MosisUnreal/Plugins/MosisSDK/README.md

MosisSDK Plugin for Unreal Engine 5.5

This plugin provides integration between Unreal Engine and MosisService, enabling a virtual smartphone display within VR/AR applications.

Overview

The MosisSDK plugin connects to the MosisService Android application via AIDL (Android Interface Definition Language) to:

• Receive rendered phone screen frames via AHardwareBuffer
• Forward touch input from VR controllers to the virtual phone
• Import hardware buffers into Vulkan for GPU-efficient rendering

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Unreal Engine Game                        │
│  ┌─────────────────────┐    ┌─────────────────────────────┐ │
│  │ UMosisPhoneComponent│◄───│     MosisVulkanTexture      │ │
│  │   (Touch Input)     │    │  (HardwareBuffer Import)    │ │
│  └──────────┬──────────┘    └──────────────▲──────────────┘ │
│             │                              │                 │
│             ▼                              │                 │
│  ┌─────────────────────────────────────────┴───────────────┐│
│  │                     MosisClient                          ││
│  │              (AIDL IMosisListener)                       ││
│  └──────────────────────────┬──────────────────────────────┘│
└─────────────────────────────┼───────────────────────────────┘
                              │ Binder IPC
┌─────────────────────────────▼───────────────────────────────┐
│                      MosisService                            │
│              (Renders phone UI via RmlUi)                    │
└─────────────────────────────────────────────────────────────┘

Prerequisites

Environment Variables

Variable	Description	Example
ANDROID_HOME	Android SDK path	C:\Users\<user>\AppData\Local\Android\Sdk
ANDROID_NDK_HOME	Android NDK path (UE5 uses 25.1.8937393)	%ANDROID_HOME%\ndk\25.1.8937393

Required SDK Components

• Android SDK Platform 36 (for AIDL binder headers)
• Android Build Tools 36.1.0 (for AIDL compiler)
• Android NDK 25.1.8937393 (bundled with UE5.5)

Install via Android Studio SDK Manager:

sdkmanager "platforms;android-36" "build-tools;36.1.0"

Project Structure

Plugins/MosisSDK/
├── Source/MosisSDK/
│   ├── MosisSDK.Build.cs           # Build configuration
│   ├── MosisSDK_UPL.xml            # Android packaging rules
│   ├── AIDL/                       # AIDL interface definitions
│   │   └── com/omixlab/mosis/
│   │       ├── IMosisService.aidl
│   │       └── IMosisListener.aidl
│   ├── Generated/                  # Auto-generated headers
│   │   ├── android/
│   │   │   └── hardware_buffer_aidl.h  # Local NDK header copy
│   │   └── aidl/com/omixlab/mosis/
│   │       ├── IMosisService.h
│   │       ├── IMosisListener.h
│   │       ├── BnMosisListener.h
│   │       └── BpMosisService.h
│   ├── Public/
│   │   ├── MosisSDK.h              # Module interface
│   │   ├── MosisPhoneComponent.h   # UE5 component
│   │   └── MosisPhoneActor.h       # Blueprint actor
│   └── Private/
│       ├── MosisSDK.cpp            # Module + JNI callbacks
│       ├── MosisClient.h/cpp       # AIDL client implementation
│       ├── MosisVulkanTexture.h/cpp # Vulkan HardwareBuffer import
│       ├── MosisPhoneComponent.cpp
│       ├── MosisPhoneActor.cpp
│       └── Android/
│           └── Generated/          # AIDL-generated .cpp (Android only)
│               └── com/omixlab/mosis/
│                   ├── IMosisService.cpp
│                   └── IMosisListener.cpp
└── Binaries/
    └── Win64/                      # Windows editor binaries
    └── Android/                    # Android binaries

Build Instructions

Windows Editor Build

Build the plugin for use in the Unreal Editor:

"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
    MosisUnrealEditor Win64 Development ^
    -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"

Output: Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll

Android Build

Build and package for Android deployment:

"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
    BuildCookRun ^
    -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
    -platform=Android ^
    -clientconfig=Development ^
    -build -cook -stage -pak -package ^
    -noP4 -utf8output

Output: Binaries/Android/MosisUnreal-arm64.apk

Clean Build

To force a full rebuild:

rmdir /s /q "Intermediate\Build"
rmdir /s /q "Binaries"
rmdir /s /q "Plugins\MosisSDK\Intermediate"
rmdir /s /q "Plugins\MosisSDK\Binaries"

Key Implementation Details

AIDL Code Generation

The Build.cs automatically runs the AIDL compiler during Android builds:

1. Input: .aidl files in Source/MosisSDK/AIDL/
2. Header Output: Source/MosisSDK/Generated/ (included for all platforms)
3. CPP Output: Source/MosisSDK/Private/Android/Generated/ (Android only)

The separation ensures Windows builds don't try to compile Android-specific AIDL bindings.

Platform-Specific Dependencies

Dependency	Windows	Android	Purpose
Core, CoreUObject, Engine	✓	✓	UE5 fundamentals
Slate, SlateCore	✓	✓	UI framework
RenderCore, RHI	✓	✓	Rendering abstraction
Launch, ApplicationCore	✗	✓	Android JNI access
Vulkan	✗	✓	Hardware buffer import
binder_ndk, android, nativewindow	✗	✓	Android system libs

NDK Header Compatibility

UE5.5 uses NDK 25, but hardware_buffer_aidl.h was added in NDK 26+. The plugin includes a local copy with modifications:

1. Removed __INTRODUCED_IN(34) annotations (causes API level errors)
2. Added __attribute__((weak)) to parcel functions (allows linking on API 33)

MosisClient (IMosisListener Implementation)

The MosisClient class implements the AIDL listener interface:

class MosisClient : public aidl::com::omixlab::mosis::BnMosisListener
{
public:
    // Called when service initializes
    ndk::ScopedAStatus onServiceInitialized(bool success) override;

    // Called when a new hardware buffer is available
    ndk::ScopedAStatus onBufferAvailable(const HardwareBuffer& buffer) override;

    // Called each frame when content updates
    ndk::ScopedAStatus onFrameAvailable() override;

    // Touch forwarding to service
    void SendTouchDown(float x, float y);
    void SendTouchMove(float x, float y);
    void SendTouchUp(float x, float y);
};

Service Connection Flow

1. FMosisSDKModule::StartupModule() calls Java MyKotlinPlugin.StartMosisService()
2. Kotlin code binds to MosisService
3. On connection, Kotlin calls native serviceConnected(IBinder binder)
4. JNI callback creates MosisClient from the binder
5. Client calls initOS() on the service
6. Service sends onBufferAvailable() with the shared hardware buffer
7. Client imports buffer into Vulkan texture
8. Service sends onFrameAvailable() each rendered frame

Usage in Blueprints

MosisPhoneActor

Drop AMosisPhoneActor into your level for a ready-to-use phone display:

1. Add to level via Place Actors > MosisPhoneActor
2. Configure material and mesh as needed
3. The actor automatically connects to MosisService on BeginPlay

MosisPhoneComponent

For custom actors, add UMosisPhoneComponent:

UPROPERTY(VisibleAnywhere)
UMosisPhoneComponent* PhoneComponent;

// In constructor
PhoneComponent = CreateDefaultSubobject<UMosisPhoneComponent>(TEXT("Phone"));

Touch Input

Send touch events from VR controller raycasts:

// In your VR interaction component
FVector2D UV = CalculateHitUV(HitResult);

if (TriggerPressed && !WasTriggerPressed)
    PhoneComponent->SendTouch(UV, EMosisTouchType::Down);
else if (TriggerPressed)
    PhoneComponent->SendTouch(UV, EMosisTouchType::Move);
else if (!TriggerPressed && WasTriggerPressed)
    PhoneComponent->SendTouch(UV, EMosisTouchType::Up);

Troubleshooting

Build Errors

"Cannot open include file: 'aidl/com/omixlab/mosis/IMosisListener.h'"

• Ensure Android SDK Platform 36 is installed
• Check ANDROID_HOME environment variable
• Run Android build first to generate headers

"LINK : fatal error LNK1181: cannot open input file 'UnrealEditor.lib'"

• The Launch module was incorrectly in global dependencies
• Should only be in Android-specific dependencies
• Solution: Move Launch to the Android platform block in Build.cs

"'Android/AndroidJNI.h' file not found"

• Add Launch and ApplicationCore to Android dependencies in Build.cs

"Cannot open include file: 'android/hardware_buffer_aidl.h'"

• This header is only in NDK 26+, but UE5.5 uses NDK 25
• Solution: Local copy in Generated/android/hardware_buffer_aidl.h

"'AHardwareBuffer_readFromParcel' is unavailable" (API level errors)

• Functions marked as API 34+ but building for API 33
• Solution: Add __attribute__((weak)) to function declarations in local header

AIDL cpp files compiled on Windows causing errors

• AIDL-generated .cpp files contain Android-only headers
• Solution: Output cpp to Private/Android/Generated/ (only compiled on Android)
• Headers go to Generated/ (included but not compiled)

Runtime Issues

Service not connecting

• Ensure MosisService app is installed and running
• Check logcat: adb logcat -s MosisSDK MosisTest
• Verify package name matches in MyKotlinPlugin.kt

Black texture / No frames

• Check onBufferAvailable callback is received
• Verify Vulkan extensions are supported on device
• Check hardware buffer format compatibility

Device Testing

:: Install both apps
adb install -r path\to\MosisService.apk
adb install -r Binaries\Android\MosisUnreal-arm64.apk

:: Launch service first
adb shell am start -n com.omixlab.mosis/.MainActivity

:: Launch game
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity

:: Monitor logs
adb logcat -s MosisSDK MosisTest RMLUI

Version History

• v1.0 - Initial implementation
  • AIDL client for MosisService connection
  • Vulkan HardwareBuffer import
  • UE5 component and actor for phone display
  • Touch input forwarding