MosisService/docs/GAME-ENGINES.md

# Game Engine Integrations

MosisService provides a virtual phone that game engines can display and interact with.

## Integration Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        MosisService                              │
│              (OpenGL ES rendering → AHardwareBuffer)             │
└───────────────────────────┬─────────────────────────────────────┘
                            │ Binder IPC + Shared Memory
          ┌─────────────────┴─────────────────┐
          ▼                                   ▼
┌─────────────────────┐             ┌─────────────────────┐
│    MosisUnreal      │             │      MosisVR        │
│   (UE5.5 Plugin)    │             │  (Unity Package)    │
│   Vulkan Import     │             │  Vulkan/OpenGL      │
└─────────────────────┘             └─────────────────────┘
```

## MosisUnreal (Unreal Engine 5.5)

**Location**: `D:\Dev\Mosis\MosisUnreal\Plugins\MosisSDK\`

### Build Script (Recommended)

Use the build script for one-command builds and deployment:

```batch
cd D:\Dev\Mosis\MosisUnreal

:: Build, install, and launch (default)
build-unreal.bat

:: Build APK only
build-unreal.bat build

:: Install to device only (requires prior build)
build-unreal.bat install

:: Launch app (starts MosisService first, then MosisUnreal)
build-unreal.bat launch

:: Clean build artifacts
build-unreal.bat clean
```

The script auto-detects connected Android devices, handles APK and OBB installation, and properly sequences app launches (MosisService must start before MosisUnreal).

### Manual Build Commands

```batch
:: Windows Editor Build
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
    MosisUnrealEditor Win64 Development ^
    -Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"

:: Android APK Build
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
    BuildCookRun ^
    -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
    -platform=Android -clientconfig=Development ^
    -build -cook -stage -pak -package -noP4

:: Clean Build (delete these folders first)
rmdir /s /q "Intermediate\Build"
rmdir /s /q "Binaries"
```

### Output Files

| Build | Output |
|-------|--------|
| Windows Editor | `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll` |
| Android APK | `Binaries/Android/MosisUnreal-arm64.apk` |
| Android OBB | `Binaries/Android/main.1.com.omixlab.MosisUnreal.obb` |

### Requirements

- Android SDK Platform 36 (for AIDL binder headers)
- Android Build Tools 36.1.0 (for AIDL compiler)
- `ANDROID_HOME` environment variable set

### Quest Deployment

**IMPORTANT**: Git Bash on Windows converts Unix paths to Windows paths. Use `MSYS_NO_PATHCONV=1` prefix for all adb push commands.

```bash
# Set Quest device ID (check with: adb devices -l)
QUEST=2G0YC5ZF7W01T0

# Install APK
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk

# Create OBB temp directory
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"

# Push OBB to temp location (MUST use MSYS_NO_PATHCONV=1)
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/

# Move OBB to final location
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal"
adb -s $QUEST shell "mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"

# Start MosisService first, then MosisUnreal
adb -s $QUEST shell am start -n com.omixlab.mosis/.MainActivity
sleep 3
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity

# Monitor logs
adb -s $QUEST logcat -s MosisSDK MosisOS MosisTest
```

**Full rebuild and deploy sequence**:
```bash
# 1. Build Android APK+OBB
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun \
    -project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" \
    -platform=Android -clientconfig=Development \
    -build -cook -stage -pak -package -noP4

# 2. Stop app, push APK and OBB, restart
QUEST=2G0YC5ZF7W01T0
adb -s $QUEST shell am force-stop com.omixlab.MosisUnreal
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal && mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
```

**Common issues**:
- App stuck on loading screen: OBB not pushed or wrong version
- `adb push` path errors: Missing `MSYS_NO_PATHCONV=1` prefix
- Service not connecting: Start `com.omixlab.mosis` before the game

### VR Pointer Interaction

The MosisSDK includes `UMosisPointerComponent` for VR ray-based touch interaction.

**Key Files**:
- `Public/MosisPointerComponent.h` - Component header
- `Private/MosisPointerComponent.cpp` - Implementation

**Features**:
- Raycast from component transform (attach as child of motion controller)
- Automatic detection of `AMosisPhoneActor` hits
- Touch state machine: Down/Move/Up events
- Optional Enhanced Input binding via `TriggerAction` property
- Manual control via `SetTriggerPressed(bool)`
- Debug ray visualization

**Blueprint Setup**:
1. Add `MosisPointerComponent` as child of `MotionControllerComponent`
2. Set `TriggerAction` to your trigger input action (optional)
3. Touch events are sent automatically when pointing at phone and triggering

**C++ Setup**:
```cpp
// In VR Pawn header
UPROPERTY(VisibleAnywhere)
UMosisPointerComponent* RightPointer;

// In constructor
RightPointer = CreateDefaultSubobject<UMosisPointerComponent>(TEXT("RightPointer"));
RightPointer->SetupAttachment(RightMotionController);

// In BeginPlay (if using Enhanced Input)
RightPointer->TriggerAction = TriggerInputAction;
```

**Manual Control (without Enhanced Input)**:
```cpp
// In your input handling code
void AMyVRPawn::OnTriggerPressed()
{
    RightPointer->SetTriggerPressed(true);
}

void AMyVRPawn::OnTriggerReleased()
{
    RightPointer->SetTriggerPressed(false);
}
```

**Component Properties**:

| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `RayLength` | float | 500.0 | Maximum ray length in cm |
| `bShowDebugRay` | bool | false | Draw debug visualization |
| `DebugRayColor` | FLinearColor | Red | Ray color when not hitting |
| `DebugRayHitColor` | FLinearColor | Green | Ray color when hitting phone |
| `TraceChannel` | ECollisionChannel | Visibility | Collision channel for raycast |
| `TriggerAction` | UInputAction* | nullptr | Enhanced Input action for trigger |

**Blueprint Functions**:

| Function | Returns | Description |
|----------|---------|-------------|
| `IsPointingAtPhone()` | bool | True if ray hits a phone actor |
| `GetTargetPhone()` | AMosisPhoneActor* | Currently targeted phone (or nullptr) |
| `GetHitLocation()` | FVector | World location of ray hit |
| `GetRayOrigin()` | FVector | Ray start position |
| `GetRayDirection()` | FVector | Ray direction vector |
| `IsTriggerPressed()` | bool | Current trigger state |
| `SetTriggerPressed(bool)` | void | Manual trigger control |

**Touch State Machine**:
```
Idle ──[raycast hit]──► Hover ──[trigger]──► Pressed
  ▲                       │                      │
  │                       │ raycast miss         │ trigger release
  │                       ▼                      ▼
  └──────────────────── Idle ◄────────────────────

Events:
- Idle → Pressed: SendTouch(Down)
- Pressed + moving: SendTouch(Move)
- Pressed → Idle: SendTouch(Up)
```

## MosisVR (Unity 6000.3.2f1)

**Location**: `D:\Dev\Mosis\MosisVR\Packages\com.omixlab.mosis_sdk\`

### Direct APK Build (Recommended)

```batch
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidDirectCI ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk"
```

### Export + Gradle Build

For more control, export a Gradle project then build separately:

```batch
:: Step 1: Export from Unity
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
    -batchmode -quit -nographics ^
    -projectPath "D:\Dev\Mosis\MosisVR" ^
    -executeMethod BuildScript.BuildAndroidCI ^
    -export true ^
    -outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR"

:: Step 2: Build with Gradle
cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
gradle assembleRelease
:: APK at: launcher\build\outputs\apk\release\launcher-release.apk
```

### Unity Editor Manual Build

1. File > Build Settings > Android
2. Player Settings: IL2CPP, ARM64, Vulkan + OpenGLES3
3. For direct APK: Uncheck "Export Project", click Build
4. For export: Check "Export Project", click Export

### Native Plugin Build (Manual)

The native plugin builds automatically via CMake during Unity's build. To rebuild manually:

```batch
cd Packages/com.omixlab.mosis_sdk/Plugins/Android/cpp
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
    -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29
cmake --build build
```

## Device Testing (Both Engines)

```bash
# Install MosisService first
adb install -r MosisService-debug.apk

# Install game client
adb install -r MosisUnreal-arm64.apk  # or MosisVR.apk

# Launch service
adb shell am start -n com.omixlab.mosis/.MainActivity

# Launch client
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
# or for Unity:
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity

# Monitor all Mosis logs
adb logcat -s MosisSDK MosisTest RMLUI Vulkan
```

## Vulkan HardwareBuffer Import

Both game engines use Vulkan to import AHardwareBuffer from MosisService.

### UE5 Implementation

MosisUnreal uses UE5's built-in Vulkan RHI API for hardware buffer import:

```cpp
#include "IVulkanDynamicRHI.h"

// Import AHardwareBuffer as Vulkan texture (zero-copy)
IVulkanDynamicRHI* VulkanRHI = GetIVulkanDynamicRHI();
FTextureRHIRef ImportedTexture = VulkanRHI->RHICreateTexture2DFromAndroidHardwareBuffer(Buffer);

// GPU-to-GPU copy to destination texture each frame
ENQUEUE_RENDER_COMMAND(CopyMosisTexture)(
    [SrcTexture, DstTexture, Width, Height](FRHICommandListImmediate& RHICmdList)
    {
        RHICmdList.Transition(FRHITransitionInfo(SrcTexture, ERHIAccess::Unknown, ERHIAccess::CopySrc));
        RHICmdList.Transition(FRHITransitionInfo(DstTexture, ERHIAccess::Unknown, ERHIAccess::CopyDest));

        FRHICopyTextureInfo CopyInfo;
        CopyInfo.Size = FIntVector(Width, Height, 1);
        RHICmdList.CopyTexture(SrcTexture, DstTexture, CopyInfo);

        RHICmdList.Transition(FRHITransitionInfo(DstTexture, ERHIAccess::CopyDest, ERHIAccess::SRVMask));
    }
);
```

**Key files:**
- `MosisPhoneTexture.cpp` - Vulkan import and GPU copy
- `MosisPhoneComponent.cpp` - Frame update coordination

### Required Vulkan Extensions

UE5's VulkanRHI enables these automatically on Android:

```
VK_ANDROID_external_memory_android_hardware_buffer
VK_KHR_external_memory
VK_KHR_dedicated_allocation
```

### Raw Vulkan Import Pattern (Reference)

For custom implementations outside UE5:

```cpp
// 1. Query buffer properties
VkAndroidHardwareBufferPropertiesANDROID props = {
    .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID
};
VkAndroidHardwareBufferFormatPropertiesANDROID formatProps = {
    .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID
};
props.pNext = &formatProps;
vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &props);

// 2. Create image with external memory
VkExternalMemoryImageCreateInfo extInfo = {
    .sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO,
    .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID
};

VkImageCreateInfo imageInfo = {
    .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
    .pNext = &extInfo,
    .imageType = VK_IMAGE_TYPE_2D,
    .format = formatProps.format,
    .extent = {desc.width, desc.height, 1},
    .mipLevels = 1,
    .arrayLayers = 1,
    .samples = VK_SAMPLE_COUNT_1_BIT,
    .tiling = VK_IMAGE_TILING_OPTIMAL,
    .usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT,
    .sharingMode = VK_SHARING_MODE_EXCLUSIVE
};
vkCreateImage(device, &imageInfo, nullptr, &image);

// 3. Import memory from HardwareBuffer
VkImportAndroidHardwareBufferInfoANDROID importInfo = {
    .sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID,
    .buffer = buffer
};

VkMemoryDedicatedAllocateInfo dedicatedInfo = {
    .sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO,
    .pNext = &importInfo,
    .image = image
};

VkMemoryAllocateInfo allocInfo = {
    .sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
    .pNext = &dedicatedInfo,
    .allocationSize = props.allocationSize,
    .memoryTypeIndex = FindMemoryType(props.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)
};
vkAllocateMemory(device, &allocInfo, nullptr, &memory);
vkBindImageMemory(device, image, memory, 0);
```

### Double Buffering

The imported image is copied to a local texture each frame via GPU copy to prevent data races with MosisService rendering. This is a fast GPU-to-GPU blit operation.