move docs to docs/ folder, merge architecture files, update references
This commit is contained in:
348
docs/GAME-ENGINES.md
Normal file
348
docs/GAME-ENGINES.md
Normal file
@@ -0,0 +1,348 @@
|
||||
# Game Engine Integrations
|
||||
|
||||
MosisService provides a virtual phone that game engines can display and interact with.
|
||||
|
||||
## Integration Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MosisService │
|
||||
│ (OpenGL ES rendering → AHardwareBuffer) │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│ Binder IPC + Shared Memory
|
||||
┌─────────────────┴─────────────────┐
|
||||
▼ ▼
|
||||
┌─────────────────────┐ ┌─────────────────────┐
|
||||
│ MosisUnreal │ │ MosisVR │
|
||||
│ (UE5.5 Plugin) │ │ (Unity Package) │
|
||||
│ Vulkan Import │ │ Vulkan/OpenGL │
|
||||
└─────────────────────┘ └─────────────────────┘
|
||||
```
|
||||
|
||||
## MosisUnreal (Unreal Engine 5.5)
|
||||
|
||||
**Location**: `D:\Dev\Mosis\MosisUnreal\Plugins\MosisSDK\`
|
||||
|
||||
### Build Commands
|
||||
|
||||
```batch
|
||||
:: Windows Editor Build
|
||||
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\Build.bat" ^
|
||||
MosisUnrealEditor Win64 Development ^
|
||||
-Project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject"
|
||||
|
||||
:: Android APK Build
|
||||
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" ^
|
||||
BuildCookRun ^
|
||||
-project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" ^
|
||||
-platform=Android -clientconfig=Development ^
|
||||
-build -cook -stage -pak -package -noP4
|
||||
|
||||
:: Clean Build (delete these folders first)
|
||||
rmdir /s /q "Intermediate\Build"
|
||||
rmdir /s /q "Binaries"
|
||||
```
|
||||
|
||||
### Output Files
|
||||
|
||||
| Build | Output |
|
||||
|-------|--------|
|
||||
| Windows Editor | `Plugins/MosisSDK/Binaries/Win64/UnrealEditor-MosisSDK.dll` |
|
||||
| Android APK | `Binaries/Android/MosisUnreal-arm64.apk` |
|
||||
| Android OBB | `Binaries/Android/main.1.com.omixlab.MosisUnreal.obb` |
|
||||
|
||||
### Requirements
|
||||
|
||||
- Android SDK Platform 36 (for AIDL binder headers)
|
||||
- Android Build Tools 36.1.0 (for AIDL compiler)
|
||||
- `ANDROID_HOME` environment variable set
|
||||
|
||||
### Quest Deployment
|
||||
|
||||
**IMPORTANT**: Git Bash on Windows converts Unix paths to Windows paths. Use `MSYS_NO_PATHCONV=1` prefix for all adb push commands.
|
||||
|
||||
```bash
|
||||
# Set Quest device ID (check with: adb devices -l)
|
||||
QUEST=2G0YC5ZF7W01T0
|
||||
|
||||
# Install APK
|
||||
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk
|
||||
|
||||
# Create OBB temp directory
|
||||
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"
|
||||
|
||||
# Push OBB to temp location (MUST use MSYS_NO_PATHCONV=1)
|
||||
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/
|
||||
|
||||
# Move OBB to final location
|
||||
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal"
|
||||
adb -s $QUEST shell "mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"
|
||||
|
||||
# Start MosisService first, then MosisUnreal
|
||||
adb -s $QUEST shell am start -n com.omixlab.mosis/.MainActivity
|
||||
sleep 3
|
||||
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
||||
|
||||
# Monitor logs
|
||||
adb -s $QUEST logcat -s MosisSDK MosisOS MosisTest
|
||||
```
|
||||
|
||||
**Full rebuild and deploy sequence**:
|
||||
```bash
|
||||
# 1. Build Android APK+OBB
|
||||
"D:\Epic\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun \
|
||||
-project="D:\Dev\Mosis\MosisUnreal\MosisUnreal.uproject" \
|
||||
-platform=Android -clientconfig=Development \
|
||||
-build -cook -stage -pak -package -noP4
|
||||
|
||||
# 2. Stop app, push APK and OBB, restart
|
||||
QUEST=2G0YC5ZF7W01T0
|
||||
adb -s $QUEST shell am force-stop com.omixlab.MosisUnreal
|
||||
adb -s $QUEST install -r D:/Dev/Mosis/MosisUnreal/Binaries/Android/MosisUnreal-arm64.apk
|
||||
adb -s $QUEST shell "mkdir -p /data/local/tmp/obb/com.omixlab.MosisUnreal"
|
||||
MSYS_NO_PATHCONV=1 adb -s $QUEST push D:/Dev/Mosis/MosisUnreal/Binaries/Android/main.1.com.omixlab.MosisUnreal.obb /data/local/tmp/obb/com.omixlab.MosisUnreal/
|
||||
adb -s $QUEST shell "rm -rf /sdcard/Android/obb/com.omixlab.MosisUnreal && mv /data/local/tmp/obb/com.omixlab.MosisUnreal /sdcard/Android/obb/"
|
||||
adb -s $QUEST shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
||||
```
|
||||
|
||||
**Common issues**:
|
||||
- App stuck on loading screen: OBB not pushed or wrong version
|
||||
- `adb push` path errors: Missing `MSYS_NO_PATHCONV=1` prefix
|
||||
- Service not connecting: Start `com.omixlab.mosis` before the game
|
||||
|
||||
### VR Pointer Interaction
|
||||
|
||||
The MosisSDK includes `UMosisPointerComponent` for VR ray-based touch interaction.
|
||||
|
||||
**Key Files**:
|
||||
- `Public/MosisPointerComponent.h` - Component header
|
||||
- `Private/MosisPointerComponent.cpp` - Implementation
|
||||
|
||||
**Features**:
|
||||
- Raycast from component transform (attach as child of motion controller)
|
||||
- Automatic detection of `AMosisPhoneActor` hits
|
||||
- Touch state machine: Down/Move/Up events
|
||||
- Optional Enhanced Input binding via `TriggerAction` property
|
||||
- Manual control via `SetTriggerPressed(bool)`
|
||||
- Debug ray visualization
|
||||
|
||||
**Blueprint Setup**:
|
||||
1. Add `MosisPointerComponent` as child of `MotionControllerComponent`
|
||||
2. Set `TriggerAction` to your trigger input action (optional)
|
||||
3. Touch events are sent automatically when pointing at phone and triggering
|
||||
|
||||
**C++ Setup**:
|
||||
```cpp
|
||||
// In VR Pawn header
|
||||
UPROPERTY(VisibleAnywhere)
|
||||
UMosisPointerComponent* RightPointer;
|
||||
|
||||
// In constructor
|
||||
RightPointer = CreateDefaultSubobject<UMosisPointerComponent>(TEXT("RightPointer"));
|
||||
RightPointer->SetupAttachment(RightMotionController);
|
||||
|
||||
// In BeginPlay (if using Enhanced Input)
|
||||
RightPointer->TriggerAction = TriggerInputAction;
|
||||
```
|
||||
|
||||
**Manual Control (without Enhanced Input)**:
|
||||
```cpp
|
||||
// In your input handling code
|
||||
void AMyVRPawn::OnTriggerPressed()
|
||||
{
|
||||
RightPointer->SetTriggerPressed(true);
|
||||
}
|
||||
|
||||
void AMyVRPawn::OnTriggerReleased()
|
||||
{
|
||||
RightPointer->SetTriggerPressed(false);
|
||||
}
|
||||
```
|
||||
|
||||
**Component Properties**:
|
||||
|
||||
| Property | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `RayLength` | float | 500.0 | Maximum ray length in cm |
|
||||
| `bShowDebugRay` | bool | false | Draw debug visualization |
|
||||
| `DebugRayColor` | FLinearColor | Red | Ray color when not hitting |
|
||||
| `DebugRayHitColor` | FLinearColor | Green | Ray color when hitting phone |
|
||||
| `TraceChannel` | ECollisionChannel | Visibility | Collision channel for raycast |
|
||||
| `TriggerAction` | UInputAction* | nullptr | Enhanced Input action for trigger |
|
||||
|
||||
**Blueprint Functions**:
|
||||
|
||||
| Function | Returns | Description |
|
||||
|----------|---------|-------------|
|
||||
| `IsPointingAtPhone()` | bool | True if ray hits a phone actor |
|
||||
| `GetTargetPhone()` | AMosisPhoneActor* | Currently targeted phone (or nullptr) |
|
||||
| `GetHitLocation()` | FVector | World location of ray hit |
|
||||
| `GetRayOrigin()` | FVector | Ray start position |
|
||||
| `GetRayDirection()` | FVector | Ray direction vector |
|
||||
| `IsTriggerPressed()` | bool | Current trigger state |
|
||||
| `SetTriggerPressed(bool)` | void | Manual trigger control |
|
||||
|
||||
**Touch State Machine**:
|
||||
```
|
||||
Idle ──[raycast hit]──► Hover ──[trigger]──► Pressed
|
||||
▲ │ │
|
||||
│ │ raycast miss │ trigger release
|
||||
│ ▼ ▼
|
||||
└──────────────────── Idle ◄────────────────────
|
||||
|
||||
Events:
|
||||
- Idle → Pressed: SendTouch(Down)
|
||||
- Pressed + moving: SendTouch(Move)
|
||||
- Pressed → Idle: SendTouch(Up)
|
||||
```
|
||||
|
||||
## MosisVR (Unity 6000.3.2f1)
|
||||
|
||||
**Location**: `D:\Dev\Mosis\MosisVR\Packages\com.omixlab.mosis_sdk\`
|
||||
|
||||
### Direct APK Build (Recommended)
|
||||
|
||||
```batch
|
||||
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
|
||||
-batchmode -quit -nographics ^
|
||||
-projectPath "D:\Dev\Mosis\MosisVR" ^
|
||||
-executeMethod BuildScript.BuildAndroidDirectCI ^
|
||||
-outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR.apk"
|
||||
```
|
||||
|
||||
### Export + Gradle Build
|
||||
|
||||
For more control, export a Gradle project then build separately:
|
||||
|
||||
```batch
|
||||
:: Step 1: Export from Unity
|
||||
"C:\Program Files\Unity\Hub\Editor\6000.3.2f1\Editor\Unity.exe" ^
|
||||
-batchmode -quit -nographics ^
|
||||
-projectPath "D:\Dev\Mosis\MosisVR" ^
|
||||
-executeMethod BuildScript.BuildAndroidCI ^
|
||||
-export true ^
|
||||
-outputPath "D:\Dev\Mosis\Builds\Unity\Android\MosisVR"
|
||||
|
||||
:: Step 2: Build with Gradle
|
||||
cd D:\Dev\Mosis\Builds\Unity\Android\MosisVR
|
||||
gradle assembleRelease
|
||||
:: APK at: launcher\build\outputs\apk\release\launcher-release.apk
|
||||
```
|
||||
|
||||
### Unity Editor Manual Build
|
||||
|
||||
1. File > Build Settings > Android
|
||||
2. Player Settings: IL2CPP, ARM64, Vulkan + OpenGLES3
|
||||
3. For direct APK: Uncheck "Export Project", click Build
|
||||
4. For export: Check "Export Project", click Export
|
||||
|
||||
### Native Plugin Build (Manual)
|
||||
|
||||
The native plugin builds automatically via CMake during Unity's build. To rebuild manually:
|
||||
|
||||
```batch
|
||||
cd Packages/com.omixlab.mosis_sdk/Plugins/Android/cpp
|
||||
cmake -B build -DCMAKE_TOOLCHAIN_FILE=%ANDROID_NDK_HOME%/build/cmake/android.toolchain.cmake ^
|
||||
-DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29
|
||||
cmake --build build
|
||||
```
|
||||
|
||||
## Device Testing (Both Engines)
|
||||
|
||||
```bash
|
||||
# Install MosisService first
|
||||
adb install -r MosisService-debug.apk
|
||||
|
||||
# Install game client
|
||||
adb install -r MosisUnreal-arm64.apk # or MosisVR.apk
|
||||
|
||||
# Launch service
|
||||
adb shell am start -n com.omixlab.mosis/.MainActivity
|
||||
|
||||
# Launch client
|
||||
adb shell am start -n com.omixlab.MosisUnreal/com.epicgames.unreal.GameActivity
|
||||
# or for Unity:
|
||||
adb shell am start -n com.omixlab.mosisvr/com.unity3d.player.UnityPlayerActivity
|
||||
|
||||
# Monitor all Mosis logs
|
||||
adb logcat -s MosisSDK MosisTest RMLUI Vulkan
|
||||
```
|
||||
|
||||
## Vulkan HardwareBuffer Import
|
||||
|
||||
Both game engines use Vulkan to import AHardwareBuffer from MosisService.
|
||||
|
||||
### Required Vulkan Extensions
|
||||
|
||||
```
|
||||
VK_ANDROID_external_memory_android_hardware_buffer
|
||||
VK_KHR_external_memory
|
||||
VK_KHR_dedicated_allocation
|
||||
```
|
||||
|
||||
### Import Pattern
|
||||
|
||||
```cpp
|
||||
// 1. Query buffer properties
|
||||
VkAndroidHardwareBufferPropertiesANDROID props = {
|
||||
.sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID
|
||||
};
|
||||
VkAndroidHardwareBufferFormatPropertiesANDROID formatProps = {
|
||||
.sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID
|
||||
};
|
||||
props.pNext = &formatProps;
|
||||
vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &props);
|
||||
|
||||
// 2. Create image with external memory
|
||||
VkExternalMemoryImageCreateInfo extInfo = {
|
||||
.sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO,
|
||||
.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID
|
||||
};
|
||||
|
||||
AHardwareBuffer_Desc desc;
|
||||
AHardwareBuffer_describe(buffer, &desc);
|
||||
|
||||
VkImageCreateInfo imageInfo = {
|
||||
.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
|
||||
.pNext = &extInfo,
|
||||
.imageType = VK_IMAGE_TYPE_2D,
|
||||
.format = formatProps.format,
|
||||
.extent = {desc.width, desc.height, 1},
|
||||
.mipLevels = 1,
|
||||
.arrayLayers = 1,
|
||||
.samples = VK_SAMPLE_COUNT_1_BIT,
|
||||
.tiling = VK_IMAGE_TILING_OPTIMAL,
|
||||
.usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT,
|
||||
.sharingMode = VK_SHARING_MODE_EXCLUSIVE
|
||||
};
|
||||
vkCreateImage(device, &imageInfo, nullptr, &image);
|
||||
|
||||
// 3. Import memory from HardwareBuffer
|
||||
VkImportAndroidHardwareBufferInfoANDROID importInfo = {
|
||||
.sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID,
|
||||
.buffer = buffer
|
||||
};
|
||||
|
||||
VkMemoryDedicatedAllocateInfo dedicatedInfo = {
|
||||
.sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO,
|
||||
.pNext = &importInfo,
|
||||
.image = image
|
||||
};
|
||||
|
||||
VkMemoryAllocateInfo allocInfo = {
|
||||
.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
|
||||
.pNext = &dedicatedInfo,
|
||||
.allocationSize = props.allocationSize,
|
||||
.memoryTypeIndex = FindMemoryType(props.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)
|
||||
};
|
||||
vkAllocateMemory(device, &allocInfo, nullptr, &memory);
|
||||
vkBindImageMemory(device, image, memory, 0);
|
||||
```
|
||||
|
||||
### Synchronization
|
||||
|
||||
Currently using CPU synchronization (`vkQueueWaitIdle` / `glFinish`). Future improvement: use Vulkan semaphores for GPU-GPU sync.
|
||||
|
||||
### Double Buffering
|
||||
|
||||
The imported image is copied to a local texture each frame to prevent data races with MosisService rendering.
|
||||
Reference in New Issue
Block a user