omar/MosisService
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1f91d7508e0ea8efde09d87465f554d6bbfcc456
MosisService/docs/GAME-ENGINES.md

12 KiB
Raw Blame History

Game Engine Integrations

MosisService provides a virtual phone that game engines can display and interact with.

Integration Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        MosisService                              │
│              (OpenGL ES rendering → AHardwareBuffer)             │
└───────────────────────────┬─────────────────────────────────────┘
                            │ Binder IPC + Shared Memory
          ┌─────────────────┴─────────────────┐
          ▼                                   ▼
┌─────────────────────┐             ┌─────────────────────┐
│    MosisUnreal      │             │      MosisVR        │
│   (UE5.5 Plugin)    │             │  (Unity Package)    │
│   Vulkan Import     │             │  Vulkan/OpenGL      │
└─────────────────────┘             └─────────────────────┘

MosisUnreal (Unreal Engine 5.5)

Location: D:\Dev\Mosis\MosisUnreal\Plugins\MosisSDK\

Build Commands

:: Windows Editor Build
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
    MosisUnrealEditor Win64 Development ^
    -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"

:: Android APK Build
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
    BuildCookRun ^
    -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
    -platform=Android -clientconfig=Development ^
    -build -cook -stage -pak -package -noP4

:: Clean Build (delete these folders first)
rmdir /s /q "Intermediate\Build"
rmdir /s /q "Binaries"

Output Files

Build Output
Windows Editor Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll
Android APK Binaries/Android/MosisUnreal-arm64.apk
Android OBB Binaries/Android/main.1.com.omixlab.MosisUnreal.obb

Requirements

  • Android SDK Platform 36 (for AIDL binder headers)
  • Android Build Tools 36.1.0 (for AIDL compiler)
  • ANDROID_HOME environment variable set

Quest Deployment

IMPORTANT: Git Bash on Windows converts Unix paths to Windows paths. Use MSYS_NO_PATHCONV=1 prefix for all adb push commands.

# Set Quest device ID (check with: adb devices -l)
QUEST=2G0YC5ZF7W01T0

# Install APK
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk

# Create OBB temp directory
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"

# Push OBB to temp location (MUST use MSYS_NO_PATHCONV=1)
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/

# Move OBB to final location
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal"
adb -s $QUEST shell "mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"

# Start MosisService first, then MosisUnreal
adb -s $QUEST shell am start -n com.omixlab.mosis/.MainActivity
sleep 3
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity

# Monitor logs
adb -s $QUEST logcat -s MosisSDK MosisOS MosisTest

Full rebuild and deploy sequence:

# 1. Build Android APK+OBB
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun \
    -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" \
    -platform=Android -clientconfig=Development \
    -build -cook -stage -pak -package -noP4

# 2. Stop app, push APK and OBB, restart
QUEST=2G0YC5ZF7W01T0
adb -s $QUEST shell am force-stop com.omixlab.MosisUnreal
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal && mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity

Common issues:

  • App stuck on loading screen: OBB not pushed or wrong version
  • adb push path errors: Missing MSYS_NO_PATHCONV=1 prefix
  • Service not connecting: Start com.omixlab.mosis before the game

VR Pointer Interaction

The MosisSDK includes UMosisPointerComponent for VR ray-based touch interaction.

Key Files:

  • Public/MosisPointerComponent.h - Component header
  • Private/MosisPointerComponent.cpp - Implementation

Features:

  • Raycast from component transform (attach as child of motion controller)
  • Automatic detection of AMosisPhoneActor hits
  • Touch state machine: Down/Move/Up events
  • Optional Enhanced Input binding via TriggerAction property
  • Manual control via SetTriggerPressed(bool)
  • Debug ray visualization

Blueprint Setup:

  1. Add MosisPointerComponent as child of MotionControllerComponent
  2. Set TriggerAction to your trigger input action (optional)
  3. Touch events are sent automatically when pointing at phone and triggering

C++ Setup:

// In VR Pawn header
UPROPERTY(VisibleAnywhere)
UMosisPointerComponent* RightPointer;

// In constructor
RightPointer = CreateDefaultSubobject<UMosisPointerComponent>(TEXT("RightPointer"));
RightPointer->SetupAttachment(RightMotionController);

// In BeginPlay (if using Enhanced Input)
RightPointer->TriggerAction = TriggerInputAction;

Manual Control (without Enhanced Input):

// In your input handling code
void AMyVRPawn::OnTriggerPressed()
{
    RightPointer->SetTriggerPressed(true);
}

void AMyVRPawn::OnTriggerReleased()
{
    RightPointer->SetTriggerPressed(false);
}

Component Properties:

Property Type Default Description
RayLength float 500.0 Maximum ray length in cm
bShowDebugRay bool false Draw debug visualization
DebugRayColor FLinearColor Red Ray color when not hitting
DebugRayHitColor FLinearColor Green Ray color when hitting phone
TraceChannel ECollisionChannel Visibility Collision channel for raycast
TriggerAction UInputAction* nullptr Enhanced Input action for trigger

Blueprint Functions:

Function Returns Description
IsPointingAtPhone() bool True if ray hits a phone actor
GetTargetPhone() AMosisPhoneActor* Currently targeted phone (or nullptr)
GetHitLocation() FVector World location of ray hit
GetRayOrigin() FVector Ray start position
GetRayDirection() FVector Ray direction vector
IsTriggerPressed() bool Current trigger state
SetTriggerPressed(bool) void Manual trigger control

Touch State Machine:

Idle ──[raycast hit]──► Hover ──[trigger]──► Pressed
  ▲                       │                      │
  │                       │ raycast miss         │ trigger release
  │                       ▼                      ▼
  └──────────────────── Idle ◄────────────────────

Events:
- Idle → Pressed: SendTouch(Down)
- Pressed + moving: SendTouch(Move)
- Pressed → Idle: SendTouch(Up)

MosisVR (Unity 6000.3.2f1)

Location: D:\Dev\Mosis\MosisVR\Packages\com.omixlab.mosis_sdk\

Direct APK Build (Recommended)

"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidDirectCI ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk"

Export + Gradle Build

For more control, export a Gradle project then build separately:

:: Step 1: Export from Unity
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidCI ^
    -export true ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR"

:: Step 2: Build with Gradle
cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
gradle assembleRelease
:: APK at: launcher\build\outputs\apk\release\launcher-release.apk

Unity Editor Manual Build

  1. File > Build Settings > Android
  2. Player Settings: IL2CPP, ARM64, Vulkan + OpenGLES3
  3. For direct APK: Uncheck "Export Project", click Build
  4. For export: Check "Export Project", click Export

Native Plugin Build (Manual)

The native plugin builds automatically via CMake during Unity's build. To rebuild manually:

cd Packages/com.omixlab.mosis_sdk/Plugins/Android/cpp
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
    -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29
cmake --build build

Device Testing (Both Engines)

# Install MosisService first
adb install -r MosisService-debug.apk

# Install game client
adb install -r MosisUnreal-arm64.apk  # or MosisVR.apk

# Launch service
adb shell am start -n com.omixlab.mosis/.MainActivity

# Launch client
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
# or for Unity:
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity

# Monitor all Mosis logs
adb logcat -s MosisSDK MosisTest RMLUI Vulkan

Vulkan HardwareBuffer Import

Both game engines use Vulkan to import AHardwareBuffer from MosisService.

Required Vulkan Extensions

VK_ANDROID_external_memory_android_hardware_buffer
VK_KHR_external_memory
VK_KHR_dedicated_allocation

Import Pattern

// 1. Query buffer properties
VkAndroidHardwareBufferPropertiesANDROID props = {
    .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID
};
VkAndroidHardwareBufferFormatPropertiesANDROID formatProps = {
    .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID
};
props.pNext = &formatProps;
vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &props);

// 2. Create image with external memory
VkExternalMemoryImageCreateInfo extInfo = {
    .sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO,
    .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID
};

AHardwareBuffer_Desc desc;
AHardwareBuffer_describe(buffer, &desc);

VkImageCreateInfo imageInfo = {
    .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
    .pNext = &extInfo,
    .imageType = VK_IMAGE_TYPE_2D,
    .format = formatProps.format,
    .extent = {desc.width, desc.height, 1},
    .mipLevels = 1,
    .arrayLayers = 1,
    .samples = VK_SAMPLE_COUNT_1_BIT,
    .tiling = VK_IMAGE_TILING_OPTIMAL,
    .usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT,
    .sharingMode = VK_SHARING_MODE_EXCLUSIVE
};
vkCreateImage(device, &imageInfo, nullptr, &image);

// 3. Import memory from HardwareBuffer
VkImportAndroidHardwareBufferInfoANDROID importInfo = {
    .sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID,
    .buffer = buffer
};

VkMemoryDedicatedAllocateInfo dedicatedInfo = {
    .sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO,
    .pNext = &importInfo,
    .image = image
};

VkMemoryAllocateInfo allocInfo = {
    .sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
    .pNext = &dedicatedInfo,
    .allocationSize = props.allocationSize,
    .memoryTypeIndex = FindMemoryType(props.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)
};
vkAllocateMemory(device, &allocInfo, nullptr, &memory);
vkBindImageMemory(device, image, memory, 0);

Synchronization

Currently using CPU synchronization (vkQueueWaitIdle / glFinish). Future improvement: use Vulkan semaphores for GPU-GPU sync.

Double Buffering

The imported image is copied to a local texture each frame to prevent data races with MosisService rendering.

Reference in New Issue View Git Blame Copy Permalink