omar/MosisService
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add developer portal planning documentation (M01-M12)

Browse Source
This commit is contained in:
omigamedev
2026-01-18 17:20:51 +01:00
parent 9805bdf175
commit 5cfee2aa66
13 changed files with 6591 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

372
DEV_PORTAL_M01_APP_PACKAGE.md Normal file
View File

@@ -0,0 +1,372 @@
# Milestone 1: App Package Format
**Status**: Planning
**Goal**: Define how apps are bundled, signed, and validated.
---
## Overview
The app package format is the foundation of the entire ecosystem. Every tool (CLI, portal, device) needs to understand this format.
---
## Questions to Answer
1. What files comprise an app package?
2. How is the manifest structured?
3. How are apps signed for integrity?
4. How are updates handled (full vs delta)?
5. What metadata is required (name, version, permissions, icons)?
---
## Package Format Options
### Option A: ZIP Archive
```
myapp.zip
├── manifest.json
├── icon.png
└── assets/
├── main.rml
├── styles.rcss
└── app.lua
```
| Pros | Cons |
|------|------|
| Simple, standard tooling | No built-in signing |
| Easy to inspect | No metadata in filename |
| Wide compatibility | Need separate signature file |
### Option B: Custom Format (.mosis)
```
myapp.mosis (binary format)
┌─────────────────────────┐
│ Magic bytes (4) │
│ Version (2) │
│ Manifest length (4) │
│ Manifest JSON │
│ Signature (256) │
│ Compressed payload │
└─────────────────────────┘
```
| Pros | Cons |
|------|------|
| Signature built-in | Custom tooling needed |
| Efficient metadata access | Harder to inspect |
| Single file | More complex implementation |
### Option C: Signed ZIP (Recommended)
```
myapp.mosis/ (actually a ZIP)
├── manifest.json
├── META-INF/
│ ├── MANIFEST.MF # File hashes
│ └── SIGNATURE.SF # Signed manifest
├── icon.png
└── assets/
└── ...
```
| Pros | Cons |
|------|------|
| Standard ZIP tooling | Slightly more complex |
| JAR/APK-style signing | |
| Easy inspection | |
| Proven approach | |
---
## Proposed Structure
```
com.developer.appname-1.0.0.mosis
├── manifest.json # App metadata
├── META-INF/
│ ├── CERT.PEM # Developer certificate
│ ├── CERT.SIG # Signature of MANIFEST.MF
│ └── MANIFEST.MF # SHA-256 hashes of all files
├── icons/
│ ├── icon-32.png
│ ├── icon-64.png
│ └── icon-128.png
├── assets/
│ ├── main.rml # Entry point
│ ├── screens/
│ │ ├── home.rml
│ │ └── settings.rml
│ ├── styles/
│ │ └── theme.rcss
│ └── scripts/
│ ├── main.lua
│ └── utils.lua
└── locales/ # Optional i18n
├── en.json
└── es.json
```
---
## Manifest Schema
### Required Fields
```json
{
"$schema": "https://mosis.dev/schemas/manifest-v1.json",
"id": "com.developer.appname",
"name": "My App",
"version": "1.0.0",
"version_code": 1,
"entry": "assets/main.rml",
"min_mosis_version": "1.0.0"
}
```
### Full Schema
```json
{
"$schema": "https://mosis.dev/schemas/manifest-v1.json",
"id": "com.developer.appname",
"name": "My App",
"version": "1.0.0",
"version_code": 1,
"description": "A short description of the app",
"entry": "assets/main.rml",
"author": {
"name": "Developer Name",
"email": "dev@example.com",
"url": "https://developer.com"
},
"permissions": [
"storage",
"network",
"camera"
],
"icons": {
"32": "icons/icon-32.png",
"64": "icons/icon-64.png",
"128": "icons/icon-128.png"
},
"min_mosis_version": "1.0.0",
"target_mosis_version": "1.2.0",
"category": "utilities",
"tags": ["productivity", "tools"],
"orientation": "portrait",
"background_color": "#FFFFFF",
"locales": ["en", "es", "fr"],
"default_locale": "en"
}
```
### Field Definitions
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `id` | string | Yes | Unique package identifier (reverse domain) |
| `name` | string | Yes | Display name (max 30 chars) |
| `version` | string | Yes | Semantic version (X.Y.Z) |
| `version_code` | integer | Yes | Incremental build number |
| `entry` | string | Yes | Path to entry RML file |
| `min_mosis_version` | string | Yes | Minimum Mosis version required |
| `description` | string | No | Short description (max 80 chars) |
| `author` | object | No | Author information |
| `permissions` | array | No | Required permissions |
| `icons` | object | No | Icon paths by size |
| `category` | string | No | App store category |
| `tags` | array | No | Searchable tags |
| `orientation` | string | No | portrait, landscape, any |
| `background_color` | string | No | Hex color for loading |
| `locales` | array | No | Supported locales |
---
## Signing Mechanism
### Algorithm
- **Key type**: Ed25519 (fast, secure, small signatures)
- **Hash**: SHA-256 for file manifests
- **Format**: PEM for keys, base64 for signatures
### MANIFEST.MF Format
```
Manifest-Version: 1.0
Created-By: mosis-cli 1.0.0
Name: manifest.json
SHA-256-Digest: base64encodedHash==
Name: assets/main.rml
SHA-256-Digest: base64encodedHash==
Name: assets/scripts/main.lua
SHA-256-Digest: base64encodedHash==
```
### Signing Flow
```
1. Generate MANIFEST.MF with SHA-256 of each file
2. Sign MANIFEST.MF with developer's Ed25519 private key
3. Store signature in META-INF/CERT.SIG
4. Include developer's public key in META-INF/CERT.PEM
```
### Verification Flow
```
1. Extract META-INF/MANIFEST.MF
2. Verify signature using CERT.PEM
3. Verify CERT.PEM is registered with developer account
4. Verify each file hash matches MANIFEST.MF
```
---
## Size Limits
| Limit | Value | Rationale |
|-------|-------|-----------|
| Max package size | 50 MB | Reasonable for mobile |
| Max individual file | 10 MB | Prevent abuse |
| Max files count | 1000 | Prevent zip bombs |
| Max path length | 256 chars | Filesystem compat |
| Max manifest size | 64 KB | Prevent abuse |
---
## Validation Rules
### Manifest Validation
- [ ] Valid JSON
- [ ] All required fields present
- [ ] `id` matches reverse domain pattern: `^[a-z][a-z0-9]*(\.[a-z][a-z0-9]*)+$`
- [ ] `version` matches semver pattern
- [ ] `version_code` is positive integer
- [ ] `entry` file exists in package
- [ ] All `permissions` are valid permission names
- [ ] All `icons` paths exist and are valid images
### Package Validation
- [ ] Valid ZIP format
- [ ] manifest.json at root
- [ ] No path traversal (../)
- [ ] No absolute paths
- [ ] No symlinks
- [ ] Under size limits
- [ ] No duplicate files
- [ ] Valid file extensions only
### Signature Validation
- [ ] MANIFEST.MF present
- [ ] CERT.SIG present
- [ ] CERT.PEM present
- [ ] Signature valid for MANIFEST.MF
- [ ] All file hashes match
- [ ] Certificate registered with developer account (for store)
---
## File Extension Rules
### Allowed Extensions
| Category | Extensions |
|----------|------------|
| UI | .rml |
| Styles | .rcss |
| Scripts | .lua |
| Images | .png, .jpg, .jpeg, .tga, .webp |
| Fonts | .ttf, .otf |
| Data | .json |
| Localization | .json (in locales/) |
| Audio | .ogg, .wav, .mp3 |
### Forbidden
- Executables: .exe, .dll, .so, .dylib
- Scripts: .sh, .bat, .ps1, .py, .js
- Archives: .zip, .tar, .gz (nested)
---
## Update Handling
### Full Update
- Download complete new package
- Verify signature
- Atomic replacement of app directory
- Preserve user data in separate location
### Delta Updates (Future)
- bsdiff-style patches
- Reduces bandwidth for minor updates
- More complex implementation
- Consider for v2
---
## Deliverables
- [ ] JSON Schema for manifest validation
- [ ] Package format specification document
- [ ] Reference implementation: package creator (Go/Rust)
- [ ] Reference implementation: package validator
- [ ] Reference implementation: signature tools
- [ ] Integration with mosis-cli
---
## Test Cases
| Test | Description |
|------|-------------|
| ValidPackage | Accept well-formed package |
| InvalidManifest | Reject malformed JSON |
| MissingRequired | Reject missing required fields |
| BadSignature | Reject invalid signature |
| TamperedFile | Reject if file hash mismatch |
| PathTraversal | Reject ../ in paths |
| OversizePackage | Reject over size limit |
| BadExtension | Reject forbidden file types |
| DuplicateFiles | Reject duplicate entries |
---
## Open Questions
1. Should we support multiple entry points (e.g., widget vs full app)?
2. Should icons be required or have defaults?
3. Delta updates in v1 or defer to v2?
4. Support for app bundles (multiple apps in one package)?
---
## References
- [Android APK format](https://developer.android.com/guide/components/fundamentals)
- [JAR signing](https://docs.oracle.com/javase/tutorial/deployment/jar/signing.html)
- [Ed25519 specification](https://ed25519.cr.yp.to/)

333
DEV_PORTAL_M02_WEB_STACK.md Normal file
View File

@@ -0,0 +1,333 @@
# Milestone 2: Web Stack Selection
**Status**: Planning
**Goal**: Choose backend technologies for the developer portal and app store API.
---
## Overview
The web stack powers the developer portal, app store API, and telemetry ingestion. This decision affects development speed, hosting costs, and long-term maintenance.
---
## Requirements
### Functional
- REST API for developer portal
- File upload handling (app packages)
- Authentication (OAuth2, API keys)
- Database integration
- Background jobs (review queue, notifications)
- WebSocket support (optional, for real-time updates)
### Non-Functional
- Handle 1000+ developers initially
- Scale to 10,000+ apps
- 99.9% uptime target
- < 200ms API response time (p95)
- Secure by default
---
## Options Analysis
### Option A: Node.js + TypeScript
#### Stack
```
Runtime: Node.js 20 LTS
Language: TypeScript
Framework: Fastify or Hono
ORM: Prisma or Drizzle
Validation: Zod
Auth: Lucia or custom JWT
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Fast development | Large ecosystem, familiar syntax |
| Type safety | TypeScript catches errors early |
| Easy hiring | Many JS/TS developers |
| Ecosystem | npm has libraries for everything |
| Hosting | Vercel, Railway, Render, any VPS |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Single-threaded | Need clustering for CPU tasks |
| Memory usage | Higher than compiled languages |
| Callback complexity | Async can get messy |
#### Example Structure
```
src/
├── index.ts
├── routes/
│ ├── auth.ts
│ ├── apps.ts
│ └── telemetry.ts
├── services/
│ ├── auth.service.ts
│ └── storage.service.ts
├── db/
│ ├── schema.ts
│ └── client.ts
└── utils/
```
#### Hosting Cost Estimate
| Service | Cost/month |
|---------|------------|
| Railway (starter) | $5-20 |
| Vercel Pro | $20 |
| VPS (4GB) | $20-40 |
---
### Option B: Go
#### Stack
```
Language: Go 1.22+
Framework: Chi, Echo, or Gin
ORM: sqlc or GORM
Validation: go-playground/validator
Auth: Custom JWT
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Performance | Fast, low memory |
| Single binary | Easy deployment |
| Concurrency | Goroutines handle load well |
| Standard library | HTTP, JSON, crypto built-in |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Verbose | Error handling boilerplate |
| Smaller ecosystem | Fewer ready-made solutions |
| Learning curve | Different paradigm |
#### Example Structure
```
cmd/
├── server/
│ └── main.go
internal/
├── api/
│ ├── handlers/
│ ├── middleware/
│ └── routes.go
├── domain/
│ ├── app.go
│ └── developer.go
├── repository/
│ └── postgres/
└── service/
```
#### Hosting Cost Estimate
| Service | Cost/month |
|---------|------------|
| Fly.io (shared) | $5-15 |
| Cloud Run | Pay per use |
| VPS (2GB) | $10-20 |
---
### Option C: Rust + Axum
#### Stack
```
Language: Rust
Framework: Axum
ORM: sqlx or SeaORM
Validation: validator crate
Auth: Custom JWT
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Maximum performance | Fastest option |
| Memory safety | No runtime errors |
| Single binary | Easy deployment |
| Long-term reliability | Compiler catches bugs |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Steep learning curve | Borrow checker takes time |
| Slower development | More code to write |
| Smaller ecosystem | Fewer web libraries |
| Compile times | Can be slow |
#### Example Structure
```
src/
├── main.rs
├── api/
│ ├── mod.rs
│ ├── handlers/
│ └── middleware/
├── domain/
├── repository/
└── service/
```
#### Hosting Cost Estimate
| Service | Cost/month |
|---------|------------|
| Fly.io | $5-10 |
| Shuttle | Free tier available |
| VPS (1GB) | $5-10 |
---
### Option D: .NET (ASP.NET Core)
#### Stack
```
Language: C#
Framework: ASP.NET Core 8
ORM: Entity Framework Core
Validation: FluentValidation
Auth: ASP.NET Identity
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Enterprise-ready | Battle-tested at scale |
| Great tooling | Visual Studio, debugging |
| Performance | Very fast (Kestrel) |
| Full-featured | Auth, validation, DI built-in |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Heavier runtime | Larger container images |
| Microsoft ecosystem | May feel locked in |
| Verbose | More boilerplate |
#### Hosting Cost Estimate
| Service | Cost/month |
|---------|------------|
| Azure App Service | $15-50 |
| VPS (4GB) | $20-40 |
---
## Evaluation Matrix
| Criteria | Weight | Node.js | Go | Rust | .NET |
|----------|--------|---------|-----|------|------|
| Dev speed | 25% | 9 | 7 | 5 | 7 |
| Performance | 20% | 6 | 9 | 10 | 8 |
| Hosting cost | 15% | 7 | 9 | 9 | 6 |
| Ecosystem | 15% | 9 | 7 | 6 | 8 |
| Team familiarity | 15% | ? | ? | ? | ? |
| Long-term maintenance | 10% | 7 | 8 | 9 | 8 |
| **Weighted Score** | | **7.4** | **7.8** | **7.2** | **7.3** |
*Team familiarity needs to be filled in based on actual experience*
---
## Framework Comparison
### Node.js Frameworks
| Framework | Req/sec | Features | Learning Curve |
|-----------|---------|----------|----------------|
| Express | 15k | Minimal, flexible | Easy |
| Fastify | 45k | Fast, schema validation | Medium |
| Hono | 50k | Ultra-light, edge-ready | Easy |
| NestJS | 20k | Full-featured, Angular-like | Steep |
### Go Frameworks
| Framework | Req/sec | Features | Learning Curve |
|-----------|---------|----------|----------------|
| net/http | 60k | Standard library | Medium |
| Chi | 55k | Lightweight router | Easy |
| Gin | 50k | Popular, middleware | Easy |
| Echo | 55k | Similar to Gin | Easy |
---
## Prototype Plan
### Phase 1: Build minimal API in top 2 choices
Endpoints to implement:
```
POST /auth/register
POST /auth/login
GET /apps
POST /apps
POST /apps/:id/versions
```
### Phase 2: Benchmark
- Requests per second
- Memory usage under load
- Development time tracking
- Code complexity comparison
### Phase 3: Decision
Based on:
- Benchmark results
- Developer experience during prototype
- Hosting cost analysis
---
## Recommendation
**Primary: Go with Chi/Echo**
- Best balance of performance and simplicity
- Low hosting costs
- Single binary deployment
- Good enough ecosystem for our needs
**Fallback: Node.js with Hono/Fastify**
- If Go feels too slow to develop in
- Larger ecosystem for edge cases
- More developers familiar with it
---
## Deliverables
- [ ] Prototype API in Go
- [ ] Prototype API in Node.js (if needed)
- [ ] Benchmark comparison document
- [ ] Final selection with rationale
- [ ] Project structure template
- [ ] Development environment setup guide
---
## Open Questions
1. Do we need GraphQL or is REST sufficient?
2. Do we need real-time features (WebSocket)?
3. What's the team's current language experience?
4. Any preference for specific cloud providers?
---
## References
- [TechEmpower Benchmarks](https://www.techempower.com/benchmarks/)
- [Go vs Node.js comparison](https://blog.logrocket.com/node-js-vs-golang/)
- [Fastify benchmarks](https://fastify.dev/benchmarks/)

418
DEV_PORTAL_M03_DATABASE.md Normal file
View File

@@ -0,0 +1,418 @@
# Milestone 3: Database Selection
**Status**: Planning
**Goal**: Choose database for developer accounts, app metadata, and analytics.
---
## Overview
The database stores all persistent data: developer accounts, app metadata, versions, telemetry events, and audit logs.
---
## Requirements
### Data Characteristics
| Data Type | Volume | Access Pattern | Consistency |
|-----------|--------|----------------|-------------|
| Developers | 10K rows | Read-heavy, low write | Strong |
| Apps | 100K rows | Read-heavy | Strong |
| Versions | 500K rows | Read-heavy | Strong |
| API Keys | 50K rows | Read-heavy | Strong |
| Telemetry | 100M+ rows | Write-heavy, append | Eventual OK |
| Audit Logs | 10M+ rows | Write-heavy, append | Eventual OK |
### Query Patterns
- Get developer by email
- List apps by developer
- Get app with latest version
- Search apps by name/tags
- Aggregate telemetry by app/day
- Time-range queries on events
---
## Options Analysis
### Option A: PostgreSQL
#### Characteristics
```
Type: Relational (SQL)
ACID: Full
JSON: Native JSONB support
Full-text: Built-in tsvector
Scaling: Vertical + read replicas
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Battle-tested | Decades of reliability |
| ACID compliance | Strong consistency |
| JSON support | JSONB for flexible data |
| Full-text search | No separate search engine needed |
| Extensions | PostGIS, pg_trgm, etc. |
| Tooling | pgAdmin, great ORMs |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Ops overhead | Need connection pooling |
| Scaling writes | Vertical scaling limits |
| Time-series | Not optimized for telemetry |
#### Hosting Options
| Provider | Free Tier | Paid |
|----------|-----------|------|
| Supabase | 500MB | $25/mo |
| Neon | 512MB | $19/mo |
| Railway | 1GB | $5/mo |
| AWS RDS | - | $15/mo+ |
| Self-hosted | - | VPS cost |
---
### Option B: SQLite + Litestream
#### Characteristics
```
Type: Embedded relational
ACID: Full
Scaling: Single writer
Backup: Litestream to S3
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Zero ops | No separate DB server |
| Fast reads | In-process, no network |
| Simple backup | Litestream handles replication |
| Low cost | Just storage costs |
| Portable | Easy local development |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Single writer | Limits write concurrency |
| No horizontal scale | One server only |
| Limited features | No full-text (without FTS5) |
#### Cost Estimate
| Component | Cost/month |
|-----------|------------|
| S3 storage (10GB) | $0.25 |
| Compute | Included in app server |
---
### Option C: PostgreSQL + TimescaleDB
#### Characteristics
```
Type: Time-series extension
Base: PostgreSQL
Scaling: Automatic partitioning
Compression: Native
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Best of both | Relational + time-series |
| Auto-partition | Handles telemetry scale |
| Compression | 90%+ compression ratio |
| Continuous aggregates | Pre-computed rollups |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Complexity | More to manage |
| Cost | Higher than plain Postgres |
| Learning curve | New concepts |
---
### Option D: Hybrid Approach
```
PostgreSQL → Developers, Apps, Versions, API Keys
ClickHouse/QuestDB → Telemetry, Analytics
Redis → Caching, Sessions
```
#### Pros
| Advantage | Details |
|-----------|---------|
| Right tool for job | Optimized for each use case |
| Scale independently | Telemetry won't affect main DB |
| Performance | Best possible for each workload |
#### Cons
| Disadvantage | Details |
|--------------|---------|
| Complexity | Multiple systems to manage |
| Cost | More infrastructure |
| Consistency | Cross-DB transactions hard |
---
## Schema Design
### Core Tables
```sql
-- Developers
CREATE TABLE developers (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email VARCHAR(255) UNIQUE NOT NULL,
name VARCHAR(100) NOT NULL,
password_hash VARCHAR(255),
oauth_provider VARCHAR(50),
oauth_id VARCHAR(255),
verified BOOLEAN DEFAULT FALSE,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- API Keys
CREATE TABLE api_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
developer_id UUID REFERENCES developers(id) ON DELETE CASCADE,
name VARCHAR(100) NOT NULL,
key_hash VARCHAR(255) NOT NULL,
key_prefix VARCHAR(10) NOT NULL, -- For display: "mk_abc..."
permissions JSONB DEFAULT '[]',
last_used_at TIMESTAMPTZ,
expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Apps
CREATE TABLE apps (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
developer_id UUID REFERENCES developers(id) ON DELETE CASCADE,
package_id VARCHAR(255) UNIQUE NOT NULL, -- com.dev.app
name VARCHAR(100) NOT NULL,
description TEXT,
category VARCHAR(50),
tags VARCHAR(50)[] DEFAULT '{}',
status VARCHAR(20) DEFAULT 'draft', -- draft, published, suspended
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- App Versions
CREATE TABLE app_versions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
app_id UUID REFERENCES apps(id) ON DELETE CASCADE,
version_code INTEGER NOT NULL,
version_name VARCHAR(20) NOT NULL,
package_url TEXT NOT NULL,
package_size BIGINT NOT NULL,
signature VARCHAR(512) NOT NULL,
permissions JSONB DEFAULT '[]',
min_mosis_version VARCHAR(20),
release_notes TEXT,
status VARCHAR(20) DEFAULT 'draft', -- draft, review, approved, published, rejected
review_notes TEXT,
published_at TIMESTAMPTZ,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(app_id, version_code)
);
-- Developer Signing Keys
CREATE TABLE signing_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
developer_id UUID REFERENCES developers(id) ON DELETE CASCADE,
name VARCHAR(100) NOT NULL,
public_key TEXT NOT NULL,
fingerprint VARCHAR(64) NOT NULL,
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMPTZ DEFAULT NOW()
);
```
### Telemetry Tables (if using PostgreSQL)
```sql
-- Telemetry Events (consider partitioning by time)
CREATE TABLE telemetry_events (
id BIGSERIAL,
app_id UUID NOT NULL,
device_id VARCHAR(64) NOT NULL, -- Hashed
event_type VARCHAR(50) NOT NULL,
event_data JSONB,
mosis_version VARCHAR(20),
timestamp TIMESTAMPTZ NOT NULL,
PRIMARY KEY (timestamp, id)
) PARTITION BY RANGE (timestamp);
-- Create monthly partitions
CREATE TABLE telemetry_events_2024_01 PARTITION OF telemetry_events
FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');
-- Crash Reports
CREATE TABLE crash_reports (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
app_id UUID NOT NULL,
app_version VARCHAR(20) NOT NULL,
device_id VARCHAR(64) NOT NULL,
crash_type VARCHAR(50) NOT NULL,
message TEXT,
stack_trace TEXT,
context JSONB,
mosis_version VARCHAR(20),
timestamp TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Daily aggregates (materialized or computed)
CREATE TABLE telemetry_daily (
app_id UUID NOT NULL,
date DATE NOT NULL,
event_type VARCHAR(50) NOT NULL,
count BIGINT NOT NULL,
unique_devices BIGINT NOT NULL,
PRIMARY KEY (app_id, date, event_type)
);
```
### Indexes
```sql
-- Developers
CREATE INDEX idx_developers_email ON developers(email);
CREATE INDEX idx_developers_oauth ON developers(oauth_provider, oauth_id);
-- Apps
CREATE INDEX idx_apps_developer ON apps(developer_id);
CREATE INDEX idx_apps_package ON apps(package_id);
CREATE INDEX idx_apps_status ON apps(status);
CREATE INDEX idx_apps_search ON apps USING gin(to_tsvector('english', name || ' ' || COALESCE(description, '')));
-- Versions
CREATE INDEX idx_versions_app ON app_versions(app_id);
CREATE INDEX idx_versions_status ON app_versions(status);
-- Telemetry
CREATE INDEX idx_telemetry_app ON telemetry_events(app_id, timestamp);
CREATE INDEX idx_telemetry_type ON telemetry_events(event_type, timestamp);
-- Crashes
CREATE INDEX idx_crashes_app ON crash_reports(app_id, timestamp);
CREATE INDEX idx_crashes_type ON crash_reports(crash_type);
```
---
## Migration Strategy
### Approach: Incremental Migrations
```
migrations/
├── 001_create_developers.sql
├── 002_create_apps.sql
├── 003_create_versions.sql
├── 004_create_telemetry.sql
└── ...
```
### Tools
- **Go**: golang-migrate, goose
- **Node.js**: Prisma Migrate, Drizzle Kit
- **Rust**: sqlx migrate, refinery
### Rollback Strategy
- Every migration has up/down
- Test rollbacks in staging
- Keep migrations small and focused
---
## Backup Strategy
### PostgreSQL
```bash
# Daily full backup
pg_dump -Fc $DATABASE_URL > backup_$(date +%Y%m%d).dump
# Continuous WAL archiving to S3
archive_command = 'aws s3 cp %p s3://backups/wal/%f'
```
### SQLite + Litestream
```yaml
# litestream.yml
dbs:
- path: /data/mosis.db
replicas:
- url: s3://backups/mosis
retention: 720h # 30 days
```
### Recovery Time Objectives
| Scenario | RTO | RPO |
|----------|-----|-----|
| Hardware failure | 1 hour | 5 minutes |
| Data corruption | 4 hours | 1 hour |
| Disaster recovery | 24 hours | 24 hours |
---
## Recommendation
### For MVP/Early Stage
**SQLite + Litestream**
- Simplest to operate
- Lowest cost
- Good enough for initial scale
- Easy migration to PostgreSQL later
### For Production Scale
**PostgreSQL + TimescaleDB**
- Handles all data types well
- Time-series for telemetry
- Proven at scale
- Good tooling ecosystem
### Hybrid (If needed later)
```
PostgreSQL → Core data (developers, apps)
TimescaleDB → Telemetry (same cluster, extension)
Redis → Caching, rate limiting
```
---
## Deliverables
- [ ] Final database selection
- [ ] Complete schema design
- [ ] Migration scripts
- [ ] Backup/restore procedures
- [ ] Connection pooling setup (if PostgreSQL)
- [ ] Monitoring queries
---
## Open Questions
1. Expected telemetry volume per day?
2. How long to retain raw telemetry?
3. Need for real-time analytics vs batch?
4. Multi-region requirements?
---
## References
- [PostgreSQL JSONB performance](https://www.postgresql.org/docs/current/datatype-json.html)
- [TimescaleDB vs InfluxDB](https://www.timescale.com/blog/timescaledb-vs-influxdb/)
- [Litestream documentation](https://litestream.io/)
- [SQLite at scale](https://www.sqlite.org/whentouse.html)

429
DEV_PORTAL_M04_AUTH.md Normal file
View File

@@ -0,0 +1,429 @@
# Milestone 4: Authentication System
**Status**: Planning
**Goal**: Secure developer authentication and app signing infrastructure.
---
## Overview
Authentication covers two areas:
1. **Developer authentication** - Login to portal, API access
2. **App signing** - Package integrity and developer verification
---
## Developer Authentication
### Methods Required
| Method | Use Case | Priority |
|--------|----------|----------|
| OAuth2 (GitHub) | Primary login | P0 |
| OAuth2 (Google) | Alternative login | P1 |
| Email + Password | Fallback | P2 |
| API Keys | CLI tools, CI/CD | P0 |
### OAuth2 Flow
```
┌─────────┐ ┌─────────┐ ┌──────────┐ ┌─────────┐
│ Browser │────►│ Portal │────►│ Provider │────►│ Callback│
└─────────┘ └─────────┘ │(GitHub) │ └────┬────┘
└──────────┘ │
▼
┌─────────────┐
│ Create/Link │
│ Account │
└─────────────┘
```
### OAuth2 Implementation
#### GitHub OAuth
```
Authorization URL: https://github.com/login/oauth/authorize
Token URL: https://github.com/login/oauth/access_token
User Info: https://api.github.com/user
Scopes: read:user, user:email
```
#### Google OAuth
```
Authorization URL: https://accounts.google.com/o/oauth2/v2/auth
Token URL: https://oauth2.googleapis.com/token
User Info: https://www.googleapis.com/oauth2/v2/userinfo
Scopes: openid, email, profile
```
### Session Management
#### JWT Tokens
```json
{
"sub": "dev_uuid",
"email": "dev@example.com",
"iat": 1704067200,
"exp": 1704153600,
"type": "access"
}
```
| Token Type | Lifetime | Storage |
|------------|----------|---------|
| Access Token | 1 hour | Memory/Cookie |
| Refresh Token | 30 days | HttpOnly Cookie |
#### Token Refresh Flow
```
1. Access token expires
2. Client sends refresh token
3. Server validates refresh token
4. Issue new access + refresh tokens
5. Invalidate old refresh token
```
### API Key Authentication
#### Key Format
```
mk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
│ │ └── 32 random bytes (base62)
│ └── Environment (live/test)
└── Prefix (mosis key)
```
#### Key Storage
```sql
-- Only store hash, never the key itself
INSERT INTO api_keys (
developer_id,
name,
key_hash, -- bcrypt or argon2
key_prefix, -- "mk_live_abc" for display
permissions
) VALUES (...);
```
#### Key Permissions
```json
{
"permissions": [
"apps:read",
"apps:write",
"versions:upload",
"telemetry:read"
]
}
```
### Rate Limiting
| Endpoint Category | Limit | Window |
|-------------------|-------|--------|
| Auth endpoints | 10 | 1 minute |
| API (authenticated) | 1000 | 1 hour |
| API (per key) | 100 | 1 minute |
| Upload | 10 | 1 hour |
---
## App Signing
### Key Generation
#### Algorithm: Ed25519
```
Private key: 32 bytes (256 bits)
Public key: 32 bytes (256 bits)
Signature: 64 bytes (512 bits)
```
#### Why Ed25519?
- Fast signing and verification
- Small key and signature sizes
- No configuration choices (secure by default)
- Widely supported
### Key Generation Flow
```bash
# Developer generates keypair locally
mosis keys generate
# Output:
# Private key saved to: ~/.mosis/signing_key.pem
# Public key saved to: ~/.mosis/signing_key.pub
#
# Fingerprint: SHA256:xxxxx
#
# IMPORTANT: Keep your private key secure!
# Upload your public key to the developer portal.
```
### Key Format
#### Private Key (PEM)
```
-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEIGxxxxx...
-----END PRIVATE KEY-----
```
#### Public Key (PEM)
```
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAxxxxx...
-----END PUBLIC KEY-----
```
### Signing Flow
```
1. Build package (ZIP all files)
2. Generate MANIFEST.MF with SHA-256 hashes
3. Sign MANIFEST.MF with private key
4. Store signature in META-INF/CERT.SIG
5. Include public key in META-INF/CERT.PEM
```
#### MANIFEST.MF Example
```
Manifest-Version: 1.0
Created-By: mosis-cli 1.0.0
Package-Id: com.developer.myapp
Version-Code: 1
Name: manifest.json
SHA-256-Digest: K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
Name: assets/main.rml
SHA-256-Digest: uU0nuZNNPgilLlLX2n2r+sSE7+N6U4DukIj3rOLvzek=
```
### Verification Flow
```
1. Extract MANIFEST.MF from package
2. Extract CERT.SIG (signature)
3. Extract CERT.PEM (public key)
4. Verify signature of MANIFEST.MF using public key
5. Verify CERT.PEM matches registered developer key
6. Verify each file hash matches MANIFEST.MF entry
```
### Key Registration
```
Developer Portal:
├── Go to Settings > Signing Keys
├── Click "Add Key"
├── Paste public key (PEM format)
├── Verify fingerprint matches local
└── Key is now registered
Server stores:
├── public_key (PEM text)
├── fingerprint (SHA256 of public key)
├── created_at
└── is_active
```
### Key Rotation
```
1. Generate new keypair
2. Register new public key in portal
3. Sign new versions with new key
4. (Optional) Revoke old key after transition period
```
### Trust Model
```
┌─────────────────────────────────────────────────┐
│ Trust Chain │
├─────────────────────────────────────────────────┤
│ │
│ Developer │
│ │ │
│ ▼ │
│ Private Key ──signs──► Package │
│ │ │
│ ▼ │
│ Public Key ──registered──► Portal │
│ │ │
│ ▼ │
│ Portal ──verifies──► Signature │
│ │ │
│ ▼ │
│ Device ──trusts──► Portal-verified packages │
│ │
└─────────────────────────────────────────────────┘
```
---
## Security Considerations
### Password Storage (if used)
```
Algorithm: Argon2id
Memory: 64 MB
Iterations: 3
Parallelism: 4
Salt: 16 bytes random
```
### Token Security
- Access tokens: Short-lived, in-memory only
- Refresh tokens: HttpOnly, Secure, SameSite=Strict
- API keys: Hashed with bcrypt, shown once on creation
### Key Security
- Private keys never leave developer's machine
- Public keys verified via fingerprint
- Key compromise: Revoke immediately, re-sign apps
### Audit Logging
```sql
INSERT INTO auth_audit_log (
developer_id,
action, -- login, logout, key_create, key_revoke
ip_address,
user_agent,
success,
failure_reason,
timestamp
) VALUES (...);
```
---
## API Endpoints
### Authentication
```
POST /auth/oauth/github # Start GitHub OAuth
GET /auth/oauth/github/callback # GitHub callback
POST /auth/oauth/google # Start Google OAuth
GET /auth/oauth/google/callback # Google callback
POST /auth/refresh # Refresh tokens
POST /auth/logout # Invalidate tokens
GET /auth/me # Get current user
```
### API Keys
```
GET /api-keys # List keys
POST /api-keys # Create key
DELETE /api-keys/:id # Revoke key
```
### Signing Keys
```
GET /signing-keys # List keys
POST /signing-keys # Register key
DELETE /signing-keys/:id # Revoke key
GET /signing-keys/:id/verify # Verify a signature
```
---
## Implementation Libraries
### Node.js
```json
{
"passport": "OAuth strategies",
"jose": "JWT handling",
"@noble/ed25519": "Ed25519 signing",
"argon2": "Password hashing"
}
```
### Go
```go
import (
"golang.org/x/oauth2"
"github.com/golang-jwt/jwt/v5"
"crypto/ed25519"
"golang.org/x/crypto/argon2"
)
```
### Rust
```toml
[dependencies]
oauth2 = "4.4"
jsonwebtoken = "9"
ed25519-dalek = "2"
argon2 = "0.5"
```
---
## Deliverables
- [ ] OAuth2 integration (GitHub)
- [ ] OAuth2 integration (Google)
- [ ] JWT token management
- [ ] API key generation and validation
- [ ] Ed25519 key generation tool
- [ ] Signature creation and verification
- [ ] Key registration API
- [ ] Audit logging
---
## Test Cases
| Test | Description |
|------|-------------|
| OAuthLogin | Complete OAuth flow |
| TokenRefresh | Refresh expired access token |
| InvalidToken | Reject tampered JWT |
| APIKeyAuth | Authenticate with API key |
| KeyGeneration | Generate valid Ed25519 keypair |
| SignPackage | Sign and verify package |
| InvalidSignature | Reject tampered package |
| KeyRevocation | Revoked key fails verification |
---
## Open Questions
1. Support for hardware security keys (YubiKey)?
2. Multi-factor authentication for portal?
3. Team accounts with role-based access?
4. Key escrow for enterprise customers?
---
## References
- [OAuth 2.0 RFC 6749](https://tools.ietf.org/html/rfc6749)
- [JWT RFC 7519](https://tools.ietf.org/html/rfc7519)
- [Ed25519 paper](https://ed25519.cr.yp.to/ed25519-20110926.pdf)
- [OWASP Authentication Cheatsheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)

464
DEV_PORTAL_M05_FRONTEND.md Normal file
View File

@@ -0,0 +1,464 @@
# Milestone 5: Developer Portal Frontend
**Status**: Planning
**Goal**: Web interface for developer account and app management.
---
## Overview
The developer portal is the primary interface for developers to manage their accounts, create apps, submit versions, and view analytics.
---
## Pages Required
### Public Pages
| Page | URL | Purpose |
|------|-----|---------|
| Landing | `/` | Marketing, sign up CTA |
| Sign In | `/login` | OAuth + password login |
| Sign Up | `/register` | Create account |
| Docs | `/docs/*` | Documentation (separate site?) |
### Authenticated Pages
| Page | URL | Purpose |
|------|-----|---------|
| Dashboard | `/dashboard` | App list, quick stats |
| App Details | `/apps/:id` | Single app overview |
| App Settings | `/apps/:id/settings` | Edit app metadata |
| App Versions | `/apps/:id/versions` | Version history |
| Submit Version | `/apps/:id/versions/new` | Upload new version |
| App Analytics | `/apps/:id/analytics` | Telemetry dashboard |
| Create App | `/apps/new` | New app wizard |
| API Keys | `/settings/keys` | Manage API keys |
| Signing Keys | `/settings/signing` | Manage signing keys |
| Profile | `/settings/profile` | Account settings |
---
## Wireframes
### Dashboard
```
┌─────────────────────────────────────────────────────────────┐
│ [Logo] Dashboard Apps Docs Settings [Avatar ▼] │
├─────────────────────────────────────────────────────────────┤
│ │
│ Welcome back, Developer! │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Total Apps │ │ Downloads │ │ Active Users│ │
│ │ 12 │ │ 45,230 │ │ 1,234 │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ Your Apps [+ New App] │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ [Icon] My Calculator v1.2.0 Published 1.2K ↓ │ │
│ │ [Icon] Notes App v2.0.1 Published 5.4K ↓ │ │
│ │ [Icon] Weather Widget v0.9.0 In Review --- │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
### App Details
```
┌─────────────────────────────────────────────────────────────┐
│ ← Back to Apps │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Icon] My Calculator │
│ com.developer.calculator │
│ ● Published │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Overview │ │ Versions │ │Analytics │ │ Settings │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ Latest Version: 1.2.0 [Submit New Version] │
│ Published: Jan 15, 2024 │
│ Downloads: 1,234 │
│ │
│ ┌────────────────────────────────────────┐ │
│ │ Downloads over time │ │
│ │ [Chart: line graph] │ │
│ └────────────────────────────────────────┘ │
│ │
│ Recent Crashes [View All →] │
│ • attempt to index nil (v1.2.0) - 23 reports │
│ • memory limit exceeded (v1.1.0) - 5 reports │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Submit Version
```
┌─────────────────────────────────────────────────────────────┐
│ Submit New Version - My Calculator │
├─────────────────────────────────────────────────────────────┤
│ │
│ Step 2 of 3: Upload Package │
│ ○ Details ● Upload ○ Review │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ ┌─────────┐ │ │
│ │ │ .mosis │ Drop your package here │ │
│ │ │ 📦 │ or click to browse │ │
│ │ └─────────┘ │ │
│ │ │ │
│ │ Max size: 50 MB │ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ ✓ Package validated │
│ ✓ Signature verified │
│ ✓ Permissions: storage, network │
│ │
│ [Back] [Continue to Review] │
│ │
└─────────────────────────────────────────────────────────────┘
```
---
## Tech Stack Options
### Option A: Next.js 14
```
Framework: Next.js 14 (App Router)
UI: shadcn/ui + Tailwind CSS
State: React Query + Zustand
Forms: React Hook Form + Zod
Charts: Recharts or Tremor
Auth: NextAuth.js
```
| Pros | Cons |
|------|------|
| SSR for SEO landing page | Complexity |
| Great developer experience | React knowledge required |
| Full-stack capability | Heavier bundle |
| Large ecosystem | |
### Option B: SvelteKit
```
Framework: SvelteKit
UI: Skeleton UI or custom
State: Svelte stores
Forms: Superforms
Charts: Chart.js or LayerCake
Auth: Lucia
```
| Pros | Cons |
|------|------|
| Fast, small bundles | Smaller ecosystem |
| Simple state management | Fewer UI libraries |
| Great DX | Less hiring pool |
### Option C: Astro + React Islands
```
Framework: Astro
UI: React components (islands)
State: Nanostores
Forms: React Hook Form
Charts: Recharts
Auth: Custom
```
| Pros | Cons |
|------|------|
| Ultra-fast static pages | More setup |
| Partial hydration | Newer approach |
| Use React where needed | Less documented patterns |
### Option D: htmx + Go Templates
```
Framework: Go templates + htmx
UI: Tailwind CSS
State: Server-side
Forms: Native HTML
Charts: Chart.js
Auth: Server sessions
```
| Pros | Cons |
|------|------|
| Simple, fast | Limited interactivity |
| No JS framework | Less polished UX |
| Server-rendered | Complex UI harder |
---
## UI Component Library
### Option A: shadcn/ui
```
Base: Radix UI primitives
Styling: Tailwind CSS
Approach: Copy-paste components
```
| Pros | Cons |
|------|------|
| High quality | React only |
| Full control | Manual updates |
| Accessible | |
### Option B: Tailwind UI
```
Base: Headless UI
Styling: Tailwind CSS
Approach: Copy-paste templates
```
| Pros | Cons |
|------|------|
| Beautiful designs | Paid ($299) |
| Production-ready | Templates, not components |
### Option C: Mantine
```
Base: Custom components
Styling: CSS-in-JS or CSS
Approach: npm package
```
| Pros | Cons |
|------|------|
| Complete solution | Opinionated |
| Many components | Larger bundle |
| Good docs | |
---
## Key Features
### File Upload
```typescript
// Drag and drop with progress
<DropZone
accept=".mosis"
maxSize={50 * 1024 * 1024}
onDrop={handleUpload}
onProgress={setProgress}
>
<UploadIcon />
<p>Drop your package here</p>
</DropZone>
```
### Real-time Validation
```typescript
// Validate package client-side before upload
async function validatePackage(file: File) {
const zip = await JSZip.loadAsync(file);
// Check manifest
const manifest = await zip.file('manifest.json')?.async('text');
if (!manifest) throw new Error('Missing manifest.json');
const parsed = JSON.parse(manifest);
ManifestSchema.parse(parsed);
// Check required files
const entry = parsed.entry;
if (!zip.file(entry)) {
throw new Error(`Entry file not found: ${entry}`);
}
return parsed;
}
```
### Analytics Dashboard
```typescript
// Recharts example
<LineChart data={downloads}>
<XAxis dataKey="date" />
<YAxis />
<Tooltip />
<Line type="monotone" dataKey="count" stroke="#8884d8" />
</LineChart>
```
---
## State Management
### Server State (React Query)
```typescript
// Fetch apps
const { data: apps, isLoading } = useQuery({
queryKey: ['apps'],
queryFn: () => api.get('/apps'),
});
// Create app mutation
const createApp = useMutation({
mutationFn: (data) => api.post('/apps', data),
onSuccess: () => {
queryClient.invalidateQueries(['apps']);
},
});
```
### Client State (Zustand)
```typescript
// UI state
const useStore = create((set) => ({
sidebarOpen: true,
toggleSidebar: () => set((s) => ({ sidebarOpen: !s.sidebarOpen })),
}));
```
---
## Authentication Flow
### Login Page
```tsx
export default function LoginPage() {
return (
<div className="min-h-screen flex items-center justify-center">
<Card className="w-96">
<CardHeader>
<h1>Sign in to Mosis</h1>
</CardHeader>
<CardContent className="space-y-4">
<Button onClick={() => signIn('github')} className="w-full">
<GitHubIcon /> Continue with GitHub
</Button>
<Button onClick={() => signIn('google')} className="w-full">
<GoogleIcon /> Continue with Google
</Button>
<Separator />
<form onSubmit={handleEmailLogin}>
<Input type="email" placeholder="Email" />
<Input type="password" placeholder="Password" />
<Button type="submit">Sign in</Button>
</form>
</CardContent>
</Card>
</div>
);
}
```
### Protected Routes
```tsx
// Middleware (Next.js)
export function middleware(request: NextRequest) {
const token = request.cookies.get('token');
if (!token && request.nextUrl.pathname.startsWith('/dashboard')) {
return NextResponse.redirect(new URL('/login', request.url));
}
}
```
---
## Responsive Design
### Breakpoints
| Size | Width | Target |
|------|-------|--------|
| sm | 640px | Mobile landscape |
| md | 768px | Tablet |
| lg | 1024px | Desktop |
| xl | 1280px | Large desktop |
### Mobile Navigation
```tsx
// Hamburger menu for mobile
<Sheet>
<SheetTrigger className="md:hidden">
<MenuIcon />
</SheetTrigger>
<SheetContent side="left">
<Navigation />
</SheetContent>
</Sheet>
```
---
## Accessibility
### Requirements
- [ ] Keyboard navigation
- [ ] Screen reader support
- [ ] Color contrast (WCAG AA)
- [ ] Focus indicators
- [ ] Alt text for images
- [ ] Form labels
- [ ] Error announcements
### Testing
```bash
# Lighthouse audit
npx lighthouse http://localhost:3000 --view
# axe-core
npm install @axe-core/react
```
---
## Deliverables
- [ ] Framework selection
- [ ] UI component library selection
- [ ] Design system (colors, typography)
- [ ] Page wireframes
- [ ] Authentication flow
- [ ] Dashboard implementation
- [ ] App management pages
- [ ] Version submission flow
- [ ] Settings pages
- [ ] Responsive design
- [ ] Accessibility audit
---
## Open Questions
1. Dark mode support?
2. Internationalization (i18n)?
3. Custom domain for docs vs integrated?
4. Email notifications UI?
---
## References
- [shadcn/ui](https://ui.shadcn.com/)
- [Next.js App Router](https://nextjs.org/docs/app)
- [Tailwind CSS](https://tailwindcss.com/)
- [React Query](https://tanstack.com/query/latest)

604
DEV_PORTAL_M06_API.md Normal file
View File

@@ -0,0 +1,604 @@
# Milestone 6: App Store Backend API
**Status**: Planning
**Goal**: REST API for app submission, review, and distribution.
---
## Overview
The backend API serves the developer portal, CLI tools, and device-side app management. It handles authentication, app lifecycle, file storage, and telemetry ingestion.
---
## API Design Principles
1. **RESTful** - Standard HTTP methods and status codes
2. **JSON** - Request and response bodies in JSON
3. **Versioned** - `/v1/` prefix for breaking changes
4. **Consistent** - Same patterns across all endpoints
5. **Documented** - OpenAPI specification
---
## Base URL
```
Production: https://api.mosis.dev/v1
Staging: https://api.staging.mosis.dev/v1
Local: http://localhost:8080/v1
```
---
## Authentication
### Headers
```
Authorization: Bearer <jwt_token>
# or
X-API-Key: mk_live_xxxxxxxx
```
### Scopes
| Scope | Description |
|-------|-------------|
| `apps:read` | Read app metadata |
| `apps:write` | Create/update apps |
| `versions:upload` | Upload new versions |
| `versions:publish` | Publish versions |
| `telemetry:read` | Read analytics |
| `keys:manage` | Manage API keys |
---
## Endpoints
### Authentication
```yaml
POST /v1/auth/oauth/github:
summary: Start GitHub OAuth flow
response: { redirect_url: string }
GET /v1/auth/oauth/github/callback:
summary: GitHub OAuth callback
query:
code: string
state: string
response: { access_token, refresh_token, user }
POST /v1/auth/oauth/google:
summary: Start Google OAuth flow
response: { redirect_url: string }
GET /v1/auth/oauth/google/callback:
summary: Google OAuth callback
POST /v1/auth/refresh:
summary: Refresh access token
body: { refresh_token: string }
response: { access_token, refresh_token }
POST /v1/auth/logout:
summary: Invalidate tokens
auth: required
GET /v1/auth/me:
summary: Get current user
auth: required
response: Developer
```
### Apps
```yaml
GET /v1/apps:
summary: List developer's apps
auth: required
query:
status: draft | published | suspended
page: number
limit: number
response: { apps: App[], total: number }
POST /v1/apps:
summary: Create new app
auth: required
body:
package_id: string # com.developer.appname
name: string
description?: string
category?: string
response: App
GET /v1/apps/:id:
summary: Get app details
auth: required
response: App
PATCH /v1/apps/:id:
summary: Update app metadata
auth: required
body:
name?: string
description?: string
category?: string
tags?: string[]
response: App
DELETE /v1/apps/:id:
summary: Delete app (if no published versions)
auth: required
response: { success: true }
```
### App Versions
```yaml
GET /v1/apps/:id/versions:
summary: List app versions
auth: required
query:
status: draft | review | approved | published | rejected
page: number
limit: number
response: { versions: AppVersion[], total: number }
POST /v1/apps/:id/versions:
summary: Create new version (get upload URL)
auth: required
body:
version_name: string # 1.0.0
version_code: number # 1
release_notes?: string
response:
version: AppVersion
upload_url: string # Presigned S3 URL
upload_expires: string # ISO timestamp
PUT /v1/apps/:id/versions/:vid/upload-complete:
summary: Mark upload as complete, trigger validation
auth: required
response: AppVersion
GET /v1/apps/:id/versions/:vid:
summary: Get version details
auth: required
response: AppVersion
POST /v1/apps/:id/versions/:vid/submit:
summary: Submit version for review
auth: required
response: AppVersion # status: review
POST /v1/apps/:id/versions/:vid/publish:
summary: Publish approved version
auth: required
response: AppVersion # status: published
DELETE /v1/apps/:id/versions/:vid:
summary: Delete draft version
auth: required
response: { success: true }
```
### Public App Store
```yaml
GET /v1/store/apps:
summary: Browse/search published apps
auth: none
query:
q: string # Search query
category: string
sort: popular | recent | name
page: number
limit: number
response: { apps: PublicApp[], total: number }
GET /v1/store/apps/:package_id:
summary: Get app store listing
auth: none
response: PublicApp
GET /v1/store/apps/:package_id/download:
summary: Get download URL for latest version
auth: none (or device token)
response:
download_url: string
version: string
size: number
signature: string
GET /v1/store/apps/:package_id/versions/:version_code/download:
summary: Get download URL for specific version
auth: none
response: { download_url, version, size, signature }
```
### API Keys
```yaml
GET /v1/keys:
summary: List API keys
auth: required
response: { keys: APIKey[] }
POST /v1/keys:
summary: Create API key
auth: required
body:
name: string
permissions: string[]
expires_at?: string
response:
key: APIKey
secret: string # Only shown once!
DELETE /v1/keys/:id:
summary: Revoke API key
auth: required
response: { success: true }
```
### Signing Keys
```yaml
GET /v1/signing-keys:
summary: List signing keys
auth: required
response: { keys: SigningKey[] }
POST /v1/signing-keys:
summary: Register signing key
auth: required
body:
name: string
public_key: string # PEM format
response: SigningKey
DELETE /v1/signing-keys/:id:
summary: Revoke signing key
auth: required
response: { success: true }
```
### Telemetry
```yaml
POST /v1/telemetry/events:
summary: Submit telemetry events (batch)
auth: device token or API key
body:
events:
- app_id: string
event_type: string
event_data: object
timestamp: string
response: { received: number }
POST /v1/telemetry/crash:
summary: Submit crash report
auth: device token or API key
body:
app_id: string
app_version: string
crash_type: string
message: string
stack_trace: string
context: object
timestamp: string
response: { id: string }
GET /v1/apps/:id/analytics:
summary: Get app analytics
auth: required
query:
start_date: string
end_date: string
metrics: downloads | active_users | crashes
response:
data:
- date: string
downloads: number
active_users: number
crashes: number
GET /v1/apps/:id/crashes:
summary: Get crash reports
auth: required
query:
version?: string
page: number
limit: number
response: { crashes: CrashReport[], total: number }
```
---
## Data Models
### Developer
```typescript
interface Developer {
id: string;
email: string;
name: string;
avatar_url?: string;
verified: boolean;
created_at: string;
updated_at: string;
}
```
### App
```typescript
interface App {
id: string;
package_id: string;
name: string;
description?: string;
category?: string;
tags: string[];
status: 'draft' | 'published' | 'suspended';
icon_url?: string;
latest_version?: AppVersion;
created_at: string;
updated_at: string;
}
```
### AppVersion
```typescript
interface AppVersion {
id: string;
app_id: string;
version_name: string;
version_code: number;
package_url?: string;
package_size?: number;
signature?: string;
permissions: string[];
min_mosis_version?: string;
release_notes?: string;
status: 'draft' | 'uploading' | 'validating' | 'review' | 'approved' | 'published' | 'rejected';
review_notes?: string;
published_at?: string;
created_at: string;
}
```
### PublicApp
```typescript
interface PublicApp {
package_id: string;
name: string;
description?: string;
category?: string;
tags: string[];
icon_url?: string;
author_name: string;
latest_version: string;
download_count: number;
rating?: number;
created_at: string;
updated_at: string;
}
```
### APIKey
```typescript
interface APIKey {
id: string;
name: string;
key_prefix: string; // "mk_live_abc..."
permissions: string[];
last_used_at?: string;
expires_at?: string;
created_at: string;
}
```
### CrashReport
```typescript
interface CrashReport {
id: string;
app_id: string;
app_version: string;
crash_type: string;
message: string;
stack_trace: string;
context: object;
occurrences: number;
first_seen: string;
last_seen: string;
}
```
---
## Error Handling
### Error Response Format
```json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Invalid package_id format",
"details": {
"field": "package_id",
"constraint": "Must match pattern: ^[a-z][a-z0-9]*(\\.[a-z][a-z0-9]*)+$"
}
}
}
```
### Error Codes
| Code | HTTP Status | Description |
|------|-------------|-------------|
| `UNAUTHORIZED` | 401 | Missing or invalid auth |
| `FORBIDDEN` | 403 | Insufficient permissions |
| `NOT_FOUND` | 404 | Resource not found |
| `VALIDATION_ERROR` | 400 | Invalid request body |
| `CONFLICT` | 409 | Resource already exists |
| `RATE_LIMITED` | 429 | Too many requests |
| `INTERNAL_ERROR` | 500 | Server error |
---
## Rate Limiting
### Headers
```
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1704067200
```
### Limits
| Endpoint Category | Limit | Window |
|-------------------|-------|--------|
| Auth | 10 | 1 minute |
| Read | 1000 | 1 hour |
| Write | 100 | 1 hour |
| Upload | 10 | 1 hour |
| Telemetry | 10000 | 1 hour |
---
## Pagination
### Request
```
GET /v1/apps?page=2&limit=20
```
### Response
```json
{
"apps": [...],
"pagination": {
"page": 2,
"limit": 20,
"total": 45,
"total_pages": 3
}
}
```
---
## Webhooks (Future)
```yaml
POST /v1/webhooks:
summary: Register webhook
body:
url: string
events: string[] # version.published, crash.new
secret: string
Webhook Payload:
headers:
X-Mosis-Signature: sha256=xxx
body:
event: string
data: object
timestamp: string
```
---
## OpenAPI Specification
Full OpenAPI 3.0 spec will be generated and hosted at:
- `https://api.mosis.dev/v1/openapi.json`
- `https://api.mosis.dev/v1/docs` (Swagger UI)
---
## Implementation Structure
```
src/
├── main.go (or index.ts)
├── api/
│ ├── routes.go
│ ├── middleware/
│ │ ├── auth.go
│ │ ├── ratelimit.go
│ │ └── logging.go
│ └── handlers/
│ ├── auth.go
│ ├── apps.go
│ ├── versions.go
│ ├── store.go
│ ├── keys.go
│ └── telemetry.go
├── service/
│ ├── app_service.go
│ ├── version_service.go
│ ├── auth_service.go
│ └── storage_service.go
├── repository/
│ ├── app_repo.go
│ ├── version_repo.go
│ └── developer_repo.go
├── domain/
│ ├── app.go
│ ├── version.go
│ └── developer.go
└── pkg/
├── validator/
└── crypto/
```
---
## Deliverables
- [ ] OpenAPI specification
- [ ] Authentication middleware
- [ ] Rate limiting middleware
- [ ] Auth endpoints
- [ ] Apps CRUD endpoints
- [ ] Versions endpoints with upload flow
- [ ] Store public endpoints
- [ ] API keys management
- [ ] Signing keys management
- [ ] Telemetry ingestion
- [ ] Error handling
- [ ] Request validation
- [ ] Integration tests
---
## Open Questions
1. GraphQL alongside REST?
2. WebSocket for real-time review status?
3. Batch operations for bulk updates?
4. API versioning strategy (URL vs header)?
---
## References
- [REST API Design Guidelines](https://github.com/microsoft/api-guidelines)
- [OpenAPI 3.0 Specification](https://swagger.io/specification/)
- [HTTP Status Codes](https://httpstatuses.com/)

451
DEV_PORTAL_M07_STORAGE.md Normal file
View File

@@ -0,0 +1,451 @@
# Milestone 7: CDN & Storage
**Status**: Planning
**Goal**: Scalable storage for app packages and assets with global distribution.
---
## Overview
Storage handles app packages, icons, screenshots, and serves downloads to devices worldwide. Must be cost-effective, fast, and reliable.
---
## Requirements
### Functional
- Store app packages (.mosis files)
- Store icons (multiple sizes)
- Store screenshots (optional)
- Serve downloads globally
- Support presigned upload URLs
- Version retention (keep old versions)
### Non-Functional
| Requirement | Target |
|-------------|--------|
| Upload speed | < 30s for 50MB |
| Download latency | < 100ms (p50) |
| Availability | 99.9% |
| Durability | 99.999999999% (11 nines) |
| Cost | Minimize egress fees |
---
## Storage Structure
```
/packages/
/{developer_id}/
/{app_id}/
/{version_code}/
package.mosis
manifest.json (extracted for quick access)
/assets/
/{app_id}/
icon-32.png
icon-64.png
icon-128.png
screenshots/
1.png
2.png
3.png
/temp/
/{upload_id}/
package.mosis (pending validation)
```
---
## Options Analysis
### Option A: Cloudflare R2
```
Storage: Object storage (S3-compatible)
CDN: Cloudflare network (automatic)
Egress: FREE (zero egress fees)
Pricing: $0.015/GB storage, $4.50/million requests
```
| Pros | Cons |
|------|------|
| No egress fees | Newer service |
| Global CDN included | Fewer regions than S3 |
| S3-compatible API | Less tooling |
| Workers integration | |
#### Cost Estimate (10K apps, 100GB)
| Component | Monthly Cost |
|-----------|--------------|
| Storage (100GB) | $1.50 |
| Requests (1M) | $4.50 |
| Egress | $0 |
| **Total** | **~$6** |
---
### Option B: AWS S3 + CloudFront
```
Storage: S3 Standard
CDN: CloudFront
Egress: $0.085-0.12/GB (varies by region)
Pricing: $0.023/GB storage
```
| Pros | Cons |
|------|------|
| Most mature | Egress costs add up |
| Best tooling | Complex pricing |
| All regions | Need CloudFront config |
| IAM integration | |
#### Cost Estimate (10K apps, 100GB, 1TB egress)
| Component | Monthly Cost |
|-----------|--------------|
| Storage (100GB) | $2.30 |
| CloudFront (1TB) | $85 |
| Requests | ~$5 |
| **Total** | **~$92** |
---
### Option C: Backblaze B2 + Cloudflare
```
Storage: Backblaze B2
CDN: Cloudflare (free egress via Bandwidth Alliance)
Egress: FREE (through Cloudflare)
Pricing: $0.005/GB storage
```
| Pros | Cons |
|------|------|
| Cheapest storage | Two services to manage |
| Free egress via CF | B2 API slightly different |
| Good reliability | Need CF proxy setup |
#### Cost Estimate (10K apps, 100GB)
| Component | Monthly Cost |
|-----------|--------------|
| Storage (100GB) | $0.50 |
| Egress (via CF) | $0 |
| Requests | ~$0.40 |
| **Total** | **~$1** |
---
### Option D: Self-hosted MinIO
```
Storage: MinIO on VPS
CDN: Cloudflare proxy
Egress: VPS bandwidth
Pricing: VPS cost only
```
| Pros | Cons |
|------|------|
| Full control | Ops overhead |
| S3-compatible | Need to manage |
| Predictable cost | Scaling complexity |
#### Cost Estimate
| Component | Monthly Cost |
|-----------|--------------|
| VPS (500GB SSD) | $20-50 |
| Cloudflare | $0 (free tier) |
| **Total** | **~$20-50** |
---
## Recommendation
**Primary: Cloudflare R2**
- Zero egress fees (biggest cost saver)
- S3-compatible (easy migration)
- Built-in global CDN
- Good enough tooling
**Fallback: Backblaze B2 + Cloudflare**
- If R2 has issues
- Even cheaper storage
- Slightly more setup
---
## Upload Flow
### Presigned URL Approach
```
┌────────┐ ┌─────────┐ ┌─────────┐
│ Client │───►│ API │───►│ R2 │
└────────┘ └─────────┘ └─────────┘
│ │ │
│ 1. Request upload URL │
│◄─────────────┤ │
│ (presigned URL) │
│ │
│ 2. Upload directly │
│────────────────────────────►│
│ │
│ 3. Notify complete │
│─────────────►│ │
│ │ 4. Validate │
│ │─────────────►│
│ │ │
│ 5. Confirm │ │
│◄─────────────┤ │
```
### API Implementation
```go
// 1. Request upload URL
func CreateVersion(c *gin.Context) {
// Create version record
version := Version{
AppID: appID,
VersionCode: req.VersionCode,
Status: "uploading",
}
db.Create(&version)
// Generate presigned URL
key := fmt.Sprintf("temp/%s/package.mosis", version.ID)
url, err := r2.PresignPut(key, 15*time.Minute)
c.JSON(200, gin.H{
"version": version,
"upload_url": url,
"expires": time.Now().Add(15 * time.Minute),
})
}
// 3. Upload complete notification
func UploadComplete(c *gin.Context) {
// Move from temp to final location
tempKey := fmt.Sprintf("temp/%s/package.mosis", versionID)
finalKey := fmt.Sprintf("packages/%s/%s/%d/package.mosis",
developerID, appID, versionCode)
r2.Copy(tempKey, finalKey)
r2.Delete(tempKey)
// Update version status
version.Status = "validating"
version.PackageURL = finalKey
// Trigger async validation
queue.Publish("validate-package", version.ID)
}
```
---
## Download Flow
### Public Downloads
```go
// Get download URL (short-lived)
func GetDownloadURL(c *gin.Context) {
version := getLatestPublishedVersion(packageID)
// Generate presigned download URL (1 hour)
url, _ := r2.PresignGet(version.PackageURL, 1*time.Hour)
c.JSON(200, gin.H{
"download_url": url,
"version": version.VersionName,
"size": version.PackageSize,
"signature": version.Signature,
})
}
```
### CDN Caching
```
Cache-Control: public, max-age=86400
```
- Packages are immutable (version code = unique)
- Cache aggressively at edge
- Invalidate only if package is pulled
---
## Icon/Screenshot Handling
### Upload
```go
// Icons uploaded with app creation/update
func UploadIcon(c *gin.Context) {
file, _ := c.FormFile("icon")
// Validate dimensions
img, _ := png.Decode(file)
if img.Bounds().Dx() != img.Bounds().Dy() {
return error("Icon must be square")
}
// Generate multiple sizes
sizes := []int{32, 64, 128}
for _, size := range sizes {
resized := resize(img, size, size)
key := fmt.Sprintf("assets/%s/icon-%d.png", appID, size)
r2.Put(key, resized)
}
}
```
### Serving
```
https://cdn.mosis.dev/assets/{app_id}/icon-64.png
```
- Public read access
- Long cache TTL (icons rarely change)
- Cloudflare image optimization (optional)
---
## Retention Policy
### Packages
| Status | Retention |
|--------|-----------|
| Published (current) | Forever |
| Published (old) | 1 year |
| Draft | 30 days |
| Rejected | 7 days |
| Failed validation | 24 hours |
### Temp Uploads
- Delete after 1 hour if not completed
- Lifecycle rule on `/temp/` prefix
### Implementation
```yaml
# R2 Lifecycle Rules
rules:
- prefix: "temp/"
expiration_days: 1
- prefix: "packages/"
tags:
status: draft
expiration_days: 30
```
---
## Backup Strategy
### R2 Built-in
- 11 nines durability
- Automatic replication within region
- No additional backup needed for packages
### Metadata Backup
- Package metadata in PostgreSQL
- PostgreSQL backups cover this
- Can regenerate URLs from DB
---
## Monitoring
### Metrics to Track
| Metric | Source |
|--------|--------|
| Upload success rate | API logs |
| Download latency | Cloudflare analytics |
| Storage usage | R2 dashboard |
| Bandwidth | R2 dashboard |
| Error rate | API logs |
### Alerts
- Upload failure rate > 5%
- Download error rate > 1%
- Storage > 80% of budget
---
## Security
### Access Control
```
Packages:
- Write: API only (presigned URLs)
- Read: Public (presigned URLs)
Assets:
- Write: API only
- Read: Public (direct CDN)
Temp:
- Write: Presigned URLs (15 min)
- Read: None (API internal only)
```
### Signed URLs
```go
// Presigned URL with expiration
url := r2.PresignPut(key, 15*time.Minute, PutOptions{
ContentType: "application/octet-stream",
ContentLength: maxSize,
})
```
---
## Deliverables
- [ ] R2 bucket setup
- [ ] Presigned URL generation
- [ ] Upload flow implementation
- [ ] Download URL generation
- [ ] Icon/screenshot upload
- [ ] Lifecycle rules for cleanup
- [ ] Monitoring dashboard
- [ ] Cost tracking
---
## Open Questions
1. Multi-region storage for lower latency?
2. Package compression (gzip/brotli)?
3. Delta updates storage structure?
4. Screenshot requirements (dimensions, count)?
---
## References
- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/)
- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
- [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html)

503
DEV_PORTAL_M08_TELEMETRY.md Normal file
View File

@@ -0,0 +1,503 @@
# Milestone 8: Telemetry System
**Status**: Planning
**Goal**: Collect app usage analytics and crash reports while respecting privacy.
---
## Overview
Telemetry provides developers with insights into app usage, performance, and crashes. Must balance usefulness with user privacy.
---
## Privacy Principles
1. **Minimal collection** - Only what's necessary
2. **No PII by default** - Anonymized device IDs
3. **Transparency** - Users know what's collected
4. **Opt-out available** - Users can disable
5. **Data retention limits** - Auto-delete old data
6. **GDPR compliance** - Export/delete on request
---
## Event Types
### Automatic Events (Default)
| Event | Description | Data |
|-------|-------------|------|
| `app_start` | App launched | version, mosis_version |
| `app_stop` | App closed | duration_seconds |
| `app_crash` | Unhandled error | crash_type, message |
| `lua_error` | Lua runtime error | message, stack (no user data) |
### Performance Events (Default)
| Event | Description | Data |
|-------|-------------|------|
| `perf_frame` | Frame time (sampled) | avg_ms, p95_ms |
| `perf_memory` | Memory usage | used_mb, limit_mb |
| `perf_startup` | Startup time | duration_ms |
### Usage Events (Opt-in)
| Event | Description | Data |
|-------|-------------|------|
| `screen_view` | Screen navigation | screen_name |
| `button_click` | UI interaction | element_id |
| `feature_used` | Feature usage | feature_name |
---
## Data Schema
### Event Payload
```json
{
"app_id": "com.developer.myapp",
"app_version": "1.2.0",
"mosis_version": "1.0.0",
"device_id": "sha256_hashed_id",
"session_id": "uuid",
"events": [
{
"type": "app_start",
"timestamp": "2024-01-15T10:30:00Z",
"data": {}
},
{
"type": "screen_view",
"timestamp": "2024-01-15T10:30:05Z",
"data": {
"screen_name": "home"
}
}
]
}
```
### Crash Report Payload
```json
{
"app_id": "com.developer.myapp",
"app_version": "1.2.0",
"mosis_version": "1.0.0",
"device_id": "sha256_hashed_id",
"timestamp": "2024-01-15T10:35:00Z",
"crash": {
"type": "lua_error",
"message": "attempt to index nil value 'user'",
"stack_trace": "main.lua:42: in function 'loadUser'\nmain.lua:15: in main chunk",
"context": {
"screen": "profile.rml",
"memory_mb": 45,
"uptime_seconds": 300
}
}
}
```
### Device ID Hashing
```lua
-- On device
local raw_id = get_android_id() -- or similar
local hashed = sha256(raw_id .. "mosis_salt_" .. app_id)
-- Result: "a3f2b1c4d5e6..."
-- Cannot reverse to original device ID
-- Different per app (can't track across apps)
```
---
## Collection Architecture
```
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Device │────►│ Batch │────►│ API │────►│ Storage │
│ │ │ Queue │ │ │ │ │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
│
│ Every 60s or
│ on app close
▼
┌──────────┐
│ Upload │
└──────────┘
```
### Client-Side Batching
```lua
-- TelemetryManager on device
local events = {}
local last_flush = os.time()
function track(event_type, data)
if not telemetry_enabled then return end
table.insert(events, {
type = event_type,
timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ"),
data = data or {}
})
-- Flush if batch is large or time elapsed
if #events >= 50 or (os.time() - last_flush) > 60 then
flush()
end
end
function flush()
if #events == 0 then return end
local payload = {
app_id = APP_ID,
app_version = APP_VERSION,
device_id = HASHED_DEVICE_ID,
events = events
}
-- Async HTTP POST
http.post(TELEMETRY_URL, json.encode(payload))
events = {}
last_flush = os.time()
end
```
---
## Storage Options
### Option A: PostgreSQL + TimescaleDB
```sql
-- Hypertable for time-series data
CREATE TABLE telemetry_events (
time TIMESTAMPTZ NOT NULL,
app_id TEXT NOT NULL,
device_id TEXT NOT NULL,
session_id TEXT,
event_type TEXT NOT NULL,
event_data JSONB,
app_version TEXT,
mosis_version TEXT
);
SELECT create_hypertable('telemetry_events', 'time');
-- Continuous aggregate for daily stats
CREATE MATERIALIZED VIEW daily_stats
WITH (timescaledb.continuous) AS
SELECT
time_bucket('1 day', time) AS day,
app_id,
event_type,
COUNT(*) as count,
COUNT(DISTINCT device_id) as unique_devices
FROM telemetry_events
GROUP BY day, app_id, event_type;
```
### Option B: ClickHouse
```sql
CREATE TABLE telemetry_events (
timestamp DateTime,
app_id String,
device_id String,
session_id String,
event_type String,
event_data String, -- JSON
app_version String,
mosis_version String
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY (app_id, timestamp);
```
### Option C: Custom + PostgreSQL
```
Raw events → Write to append-only log
Aggregator → Process hourly → Write to PostgreSQL
Cleanup → Delete raw after 24h
```
---
## Aggregation
### Pre-computed Metrics
| Metric | Granularity | Retention |
|--------|-------------|-----------|
| Daily active users | Day | 2 years |
| Event counts | Day | 1 year |
| Crash counts | Day | 1 year |
| Session duration | Day | 90 days |
| Performance percentiles | Day | 90 days |
### Aggregation Queries
```sql
-- Daily active users
SELECT
DATE_TRUNC('day', time) as day,
COUNT(DISTINCT device_id) as dau
FROM telemetry_events
WHERE app_id = $1
AND event_type = 'app_start'
AND time > NOW() - INTERVAL '30 days'
GROUP BY day
ORDER BY day;
-- Crash rate by version
SELECT
app_version,
COUNT(*) FILTER (WHERE event_type = 'app_crash') as crashes,
COUNT(*) FILTER (WHERE event_type = 'app_start') as starts,
ROUND(
100.0 * COUNT(*) FILTER (WHERE event_type = 'app_crash') /
NULLIF(COUNT(*) FILTER (WHERE event_type = 'app_start'), 0),
2
) as crash_rate
FROM telemetry_events
WHERE app_id = $1
AND time > NOW() - INTERVAL '7 days'
GROUP BY app_version;
```
---
## Crash Grouping
### Stack Trace Fingerprinting
```go
func fingerprintCrash(crash CrashReport) string {
// Normalize stack trace
normalized := normalizeStackTrace(crash.StackTrace)
// Hash key components
key := fmt.Sprintf("%s:%s:%s",
crash.CrashType,
crash.Message,
normalized,
)
return sha256(key)[:16]
}
func normalizeStackTrace(stack string) string {
// Remove line numbers (they change with code updates)
// Remove memory addresses
// Keep function names and file names
re := regexp.MustCompile(`:\d+:`)
return re.ReplaceAllString(stack, ":?:")
}
```
### Crash Groups Table
```sql
CREATE TABLE crash_groups (
id UUID PRIMARY KEY,
app_id TEXT NOT NULL,
fingerprint TEXT NOT NULL,
crash_type TEXT NOT NULL,
message TEXT,
sample_stack_trace TEXT,
first_seen TIMESTAMPTZ NOT NULL,
last_seen TIMESTAMPTZ NOT NULL,
occurrence_count INT DEFAULT 1,
affected_versions TEXT[],
status TEXT DEFAULT 'open', -- open, resolved, ignored
UNIQUE(app_id, fingerprint)
);
```
---
## Developer Dashboard
### Metrics View
```
┌─────────────────────────────────────────────────────────────┐
│ Analytics - My Calculator │
├─────────────────────────────────────────────────────────────┤
│ │
│ Date Range: [Last 30 days ▼] │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Daily Users │ │ Crashes │ │ Crash-free │ │
│ │ 1,234 │ │ 23 │ │ 98.1% │ │
│ │ ▲ +12% │ │ ▼ -45% │ │ ▲ +2% │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Daily Active Users │ │
│ │ [Line chart showing DAU over time] │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Version Distribution │ │
│ │ [Pie chart: v1.2.0: 60%, v1.1.0: 30%, v1.0.0: 10%]│ │
│ └────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Crashes View
```
┌─────────────────────────────────────────────────────────────┐
│ Crashes - My Calculator │
├─────────────────────────────────────────────────────────────┤
│ │
│ Filter: [All versions ▼] [Open ▼] │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ ● attempt to index nil value 'user' │ │
│ │ lua_error • 156 occurrences • v1.2.0 │ │
│ │ First: Jan 10 • Last: Jan 15 │ │
│ │ [View] │ │
│ ├──────────────────────────────────────────────────────┤ │
│ │ ● memory limit exceeded │ │
│ │ sandbox_error • 23 occurrences • v1.1.0, v1.2.0 │ │
│ │ First: Jan 5 • Last: Jan 14 │ │
│ │ [View] │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
---
## API Endpoints
```yaml
# Ingestion (from devices)
POST /v1/telemetry/events:
auth: device_token or api_key
body: { app_id, device_id, events[] }
response: { received: number }
POST /v1/telemetry/crash:
auth: device_token or api_key
body: { app_id, device_id, crash }
response: { id: string }
# Dashboard (for developers)
GET /v1/apps/:id/analytics/overview:
auth: required
query: { start_date, end_date }
response: { dau, crashes, crash_free_rate, ... }
GET /v1/apps/:id/analytics/events:
auth: required
query: { start_date, end_date, event_type }
response: { data: [{ date, count, unique_devices }] }
GET /v1/apps/:id/crashes:
auth: required
query: { version, status, page, limit }
response: { crashes: CrashGroup[], total }
GET /v1/apps/:id/crashes/:fingerprint:
auth: required
response: { crash_group, recent_occurrences[] }
PATCH /v1/apps/:id/crashes/:fingerprint:
auth: required
body: { status: 'resolved' | 'ignored' }
response: { crash_group }
```
---
## Data Retention
| Data Type | Retention | Reason |
|-----------|-----------|--------|
| Raw events | 7 days | Debugging |
| Daily aggregates | 2 years | Trends |
| Crash reports | 90 days | Investigation |
| Crash groups | Forever | Issue tracking |
### Cleanup Job
```sql
-- Run daily
DELETE FROM telemetry_events
WHERE time < NOW() - INTERVAL '7 days';
DELETE FROM crash_reports
WHERE timestamp < NOW() - INTERVAL '90 days';
```
---
## Privacy Controls
### User Settings
```
Settings > Privacy > Analytics
├── [✓] Send crash reports (helps developers fix bugs)
├── [ ] Send usage analytics (how you use apps)
└── [Request Data Deletion]
```
### GDPR Endpoints
```yaml
# User requests their data
GET /v1/privacy/export:
auth: user_token
response: { download_url } # JSON export of all data
# User requests deletion
DELETE /v1/privacy/data:
auth: user_token
response: { status: 'scheduled' } # Delete within 30 days
```
---
## Deliverables
- [ ] Event schema specification
- [ ] Client-side SDK for batching
- [ ] Ingestion API endpoints
- [ ] Storage setup (TimescaleDB or ClickHouse)
- [ ] Aggregation jobs
- [ ] Crash grouping logic
- [ ] Developer dashboard
- [ ] Privacy controls
- [ ] Data retention automation
- [ ] GDPR export/delete
---
## Open Questions
1. Real-time crash alerts (email/Slack)?
2. Sampling for high-volume apps?
3. Custom events API for developers?
4. Benchmarks/comparisons with similar apps?
---
## References
- [GDPR Requirements](https://gdpr.eu/)
- [TimescaleDB Best Practices](https://docs.timescale.com/timescaledb/latest/)
- [Sentry Crash Grouping](https://docs.sentry.io/product/data-management-settings/event-grouping/)

474
DEV_PORTAL_M09_REVIEW.md Normal file
View File

@@ -0,0 +1,474 @@
# Milestone 9: App Review System
**Status**: Planning
**Goal**: Automated and manual review process for app submissions.
---
## Overview
The review system ensures apps meet quality and security standards before publication. Balances automation with manual review for edge cases.
---
## Review Pipeline
```
┌─────────┐ ┌───────────┐ ┌─────────┐ ┌──────────┐ ┌───────────┐
│ Submit │──►│ Automated │──►│ Manual │──►│ Approved │──►│ Published │
│ │ │ Checks │ │ Review │ │ │ │ │
└─────────┘ └───────────┘ └─────────┘ └──────────┘ └───────────┘
│ │
▼ ▼
┌─────────┐ ┌─────────┐
│ Failed │ │Rejected │
│(auto-fix)│ │(feedback)│
└─────────┘ └─────────┘
```
---
## Automated Checks
### Tier 1: Package Validation (Blocking)
| Check | Description | Action on Fail |
|-------|-------------|----------------|
| Valid ZIP | Package is valid ZIP format | Reject |
| Manifest exists | manifest.json at root | Reject |
| Manifest valid | JSON parses, required fields | Reject |
| Signature valid | Package signed with registered key | Reject |
| Entry exists | Entry point file exists | Reject |
| Size limits | Under max package size | Reject |
| No forbidden files | No .exe, .dll, etc. | Reject |
### Tier 2: Content Validation (Blocking)
| Check | Description | Action on Fail |
|-------|-------------|----------------|
| RML valid | All RML files parse | Reject |
| RCSS valid | All RCSS files parse | Reject |
| Lua syntax | All Lua files parse | Reject |
| Icons valid | Icons are valid images | Reject |
| Icon sizes | Required icon sizes present | Reject |
| Path safety | No path traversal attempts | Reject |
### Tier 3: Security Analysis (Warning/Flag)
| Check | Description | Action on Fail |
|-------|-------------|----------------|
| Dangerous patterns | Known malicious Lua patterns | Flag for review |
| Excessive permissions | Unusual permission combos | Flag for review |
| Obfuscated code | Heavily obfuscated Lua | Flag for review |
| External URLs | Hardcoded external URLs | Flag for review |
| Large assets | Unusually large files | Warning |
### Tier 4: Quality Checks (Warning)
| Check | Description | Action on Fail |
|-------|-------------|----------------|
| Description length | Meaningful description | Warning |
| Release notes | Non-empty release notes | Warning |
| Icon quality | Not placeholder/blank | Warning |
| Localization | Locale files complete | Warning |
---
## Implementation
### Validation Worker
```go
type ValidationResult struct {
Passed bool
Errors []ValidationError
Warnings []ValidationWarning
Flags []ReviewFlag
}
type ValidationError struct {
Code string
Message string
File string
Line int
}
func ValidatePackage(packagePath string) ValidationResult {
result := ValidationResult{Passed: true}
// Tier 1: Package validation
if err := validateZip(packagePath); err != nil {
result.AddError("INVALID_ZIP", err.Error())
return result
}
manifest, err := extractManifest(packagePath)
if err != nil {
result.AddError("INVALID_MANIFEST", err.Error())
return result
}
if err := validateSignature(packagePath, manifest); err != nil {
result.AddError("INVALID_SIGNATURE", err.Error())
return result
}
// Tier 2: Content validation
files, _ := listFiles(packagePath)
for _, file := range files {
switch filepath.Ext(file) {
case ".rml":
if err := validateRML(file); err != nil {
result.AddError("INVALID_RML", err.Error(), file)
}
case ".rcss":
if err := validateRCSS(file); err != nil {
result.AddError("INVALID_RCSS", err.Error(), file)
}
case ".lua":
if err := validateLua(file); err != nil {
result.AddError("INVALID_LUA", err.Error(), file)
}
if flags := analyzeLuaSecurity(file); len(flags) > 0 {
result.Flags = append(result.Flags, flags...)
}
}
}
// Tier 3: Security analysis
if hasDangerousPatterns(files) {
result.AddFlag("DANGEROUS_PATTERNS", "Code contains suspicious patterns")
}
// Tier 4: Quality checks
if len(manifest.Description) < 10 {
result.AddWarning("SHORT_DESCRIPTION", "Description is very short")
}
result.Passed = len(result.Errors) == 0
return result
}
```
### Dangerous Pattern Detection
```go
var dangerousPatterns = []struct {
Pattern *regexp.Regexp
Reason string
}{
{
regexp.MustCompile(`loadstring\s*\(`),
"Dynamic code execution",
},
{
regexp.MustCompile(`debug\s*\.\s*\w+`),
"Debug library usage",
},
{
regexp.MustCompile(`os\s*\.\s*execute`),
"OS command execution",
},
{
regexp.MustCompile(`io\s*\.\s*\w+`),
"Direct I/O operations",
},
{
regexp.MustCompile(`ffi\s*\.\s*\w+`),
"FFI usage",
},
{
regexp.MustCompile(`package\s*\.\s*loadlib`),
"Native library loading",
},
}
func analyzeLuaSecurity(content string) []ReviewFlag {
var flags []ReviewFlag
for _, dp := range dangerousPatterns {
if dp.Pattern.MatchString(content) {
flags = append(flags, ReviewFlag{
Type: "SECURITY",
Reason: dp.Reason,
})
}
}
return flags
}
```
---
## Manual Review
### When Required
| Trigger | Reason |
|---------|--------|
| New developer | First app submission |
| Dangerous permissions | camera, microphone, contacts, location |
| Security flags | Automated checks flagged concerns |
| User reports | Existing app reported |
| Appeal | Developer contests rejection |
### Review Queue UI
```
┌─────────────────────────────────────────────────────────────┐
│ Review Queue [14 pending] │
├─────────────────────────────────────────────────────────────┤
│ │
│ Filter: [All ▼] [Flagged first ▼] │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 🚩 Weather Pro v2.0.0 │ │
│ │ com.newdev.weather • New developer │ │
│ │ Permissions: location, network │ │
│ │ Flags: First submission │ │
│ │ Submitted: 2 hours ago │ │
│ │ [Review] [Auto-approve]│ │
│ ├──────────────────────────────────────────────────────┤ │
│ │ 🚩 Photo Editor v1.5.0 │ │
│ │ com.trusted.photos • Verified developer │ │
│ │ Permissions: camera, storage │ │
│ │ Flags: DANGEROUS_PATTERNS (1) │ │
│ │ Submitted: 5 hours ago │ │
│ │ [Review] [Auto-approve]│ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Review Detail View
```
┌─────────────────────────────────────────────────────────────┐
│ Review: Weather Pro v2.0.0 │
├─────────────────────────────────────────────────────────────┤
│ │
│ Package: com.newdev.weather │
│ Developer: newdev@example.com (New - first app) │
│ Submitted: Jan 15, 2024 10:30 AM │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Automated Checks │ │
│ │ ✓ Package validation passed │ │
│ │ ✓ Content validation passed │ │
│ │ ⚠ Warning: SHORT_DESCRIPTION │ │
│ │ 🚩 Flag: First submission from new developer │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ Permissions Requested: │
│ • location (fine) - Used for weather forecasts │
│ • network - Required for API calls │
│ │
│ Files: [Expand to browse] │
│ ├── manifest.json │
│ ├── assets/ │
│ │ ├── main.rml │
│ │ └── scripts/ │
│ │ └── weather.lua [View source] │
│ └── icons/ │
│ │
│ Reviewer Notes: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ [Approve] [Reject with feedback] [Request changes] │
│ │
└─────────────────────────────────────────────────────────────┘
```
---
## Review States
```
┌─────────┐
│ Draft │
└────┬────┘
│ submit
▼
┌─────────┐
┌───────│Uploaded │
│ └────┬────┘
│ │ validation
│ ▼
│ ┌──────────┐
failed │ │Validating│
│ └────┬─────┘
│ │
│ ┌───────┴───────┐
│ │ │
▼ ▼ ▼
┌────────┐ ┌─────────┐
│ Failed │ │In Review│
└────────┘ └────┬────┘
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌────────┐ ┌──────────┐
│ Approved │ │Rejected│ │ Changes │
└────┬─────┘ └────────┘ │ Requested│
│ └──────────┘
│ publish
▼
┌──────────┐
│Published │
└──────────┘
```
---
## Rejection Feedback
### Feedback Template
```json
{
"reason": "SECURITY_CONCERN",
"message": "Your app contains patterns that raise security concerns.",
"details": [
{
"file": "scripts/main.lua",
"line": 42,
"issue": "Usage of loadstring() is not allowed",
"suggestion": "Use static Lua code instead of dynamic evaluation"
}
],
"can_resubmit": true,
"appeal_available": true
}
```
### Common Rejection Reasons
| Code | Description |
|------|-------------|
| `SECURITY_CONCERN` | Security issues found |
| `QUALITY_ISSUE` | Doesn't meet quality standards |
| `POLICY_VIOLATION` | Violates content policy |
| `METADATA_ISSUE` | Incorrect/misleading metadata |
| `PERMISSION_ABUSE` | Unnecessary permissions |
| `COPYRIGHT` | Copyright/trademark issues |
---
## Appeal Process
```
1. Developer receives rejection
2. Developer clicks "Appeal" (within 14 days)
3. Provides justification
4. Different reviewer examines
5. Final decision (approve/uphold rejection)
```
---
## Review SLA
| Submission Type | Target Time |
|-----------------|-------------|
| Auto-approved | Instant |
| New developer | 24 hours |
| Flagged | 48 hours |
| Appeal | 72 hours |
---
## API Endpoints
```yaml
# Developer endpoints
POST /v1/apps/:id/versions/:vid/submit:
summary: Submit for review
response: { version, estimated_review_time }
GET /v1/apps/:id/versions/:vid/review-status:
summary: Get review status
response: { status, feedback?, estimated_completion }
POST /v1/apps/:id/versions/:vid/appeal:
summary: Appeal rejection
body: { justification }
response: { appeal_id, status }
# Internal review endpoints (admin only)
GET /v1/admin/review-queue:
summary: Get pending reviews
response: { items[], total }
GET /v1/admin/review/:version_id:
summary: Get review details
response: { version, validation_result, flags }
POST /v1/admin/review/:version_id/approve:
summary: Approve version
body: { notes? }
response: { version }
POST /v1/admin/review/:version_id/reject:
summary: Reject version
body: { reason, message, details[] }
response: { version }
```
---
## Metrics
### Review Performance
| Metric | Target |
|--------|--------|
| Auto-approval rate | > 70% |
| Average review time | < 24 hours |
| Rejection rate | < 20% |
| Appeal overturn rate | < 10% |
### Dashboard
```sql
-- Review stats
SELECT
DATE_TRUNC('week', submitted_at) as week,
COUNT(*) as submissions,
COUNT(*) FILTER (WHERE status = 'published') as approved,
COUNT(*) FILTER (WHERE status = 'rejected') as rejected,
AVG(EXTRACT(EPOCH FROM (reviewed_at - submitted_at))/3600) as avg_hours
FROM app_versions
WHERE submitted_at > NOW() - INTERVAL '30 days'
GROUP BY week;
```
---
## Deliverables
- [ ] Validation worker implementation
- [ ] Dangerous pattern database
- [ ] Review queue UI
- [ ] Reviewer tools
- [ ] Rejection feedback system
- [ ] Appeal workflow
- [ ] Review metrics dashboard
- [ ] SLA monitoring
---
## Open Questions
1. Automated approval for trusted developers?
2. Community moderators?
3. Content policy document?
4. Rate limiting resubmissions?
---
## References
- [Apple App Review Guidelines](https://developer.apple.com/app-store/review/guidelines/)
- [Google Play Policy](https://play.google.com/about/developer-content-policy/)

603
DEV_PORTAL_M10_DEVICE.md Normal file
View File

@@ -0,0 +1,603 @@
# Milestone 10: Device-Side App Management
**Status**: Planning
**Goal**: Install, update, and manage apps on Mosis devices.
---
## Overview
Device-side app management handles the full lifecycle of apps on user devices: discovery, installation, updates, and removal.
---
## Components
```
┌─────────────────────────────────────────────────────────────┐
│ MosisService │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ AppManager │ │ UpdateService │ │ AppStore UI │ │
│ │ (C++ class) │ │ (Background) │ │ (System App) │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
│ │ │ │ │
│ └───────────────────┼───────────────────┘ │
│ │ │
│ ┌────────┴────────┐ │
│ │ LuaSandboxManager│ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
---
## Storage Layout
```
/data/mosis/
├── config/
│ ├── device.json # Device ID, settings
│ └── apps.json # Installed apps registry
├── apps/
│ └── {package_id}/
│ ├── package/ # Extracted app files
│ │ ├── manifest.json
│ │ └── assets/
│ ├── data/ # App data (VirtualFS)
│ ├── cache/ # App cache
│ └── db/ # SQLite databases
├── downloads/ # Temp download location
└── backups/ # App data backups (before update)
```
---
## AppManager Class
### Interface
```cpp
namespace mosis {
struct InstalledApp {
std::string package_id;
std::string name;
std::string version_name;
int version_code;
std::string install_path;
std::vector<std::string> permissions;
std::chrono::system_clock::time_point installed_at;
std::chrono::system_clock::time_point updated_at;
int64_t package_size;
int64_t data_size;
};
struct InstallProgress {
enum class Stage {
Downloading,
Verifying,
Extracting,
Registering,
Complete,
Failed
};
Stage stage;
float progress; // 0.0 - 1.0
std::string error;
};
using ProgressCallback = std::function<void(const InstallProgress&)>;
class AppManager {
public:
explicit AppManager(const std::string& data_root);
~AppManager();
// Installation
bool Install(const std::string& package_url,
const std::string& signature,
ProgressCallback callback);
bool InstallFromFile(const std::string& package_path,
ProgressCallback callback);
// Uninstallation
bool Uninstall(const std::string& package_id, bool keep_data = false);
// Updates
bool Update(const std::string& package_id,
const std::string& package_url,
const std::string& signature,
ProgressCallback callback);
// Queries
std::vector<InstalledApp> GetInstalledApps() const;
std::optional<InstalledApp> GetApp(const std::string& package_id) const;
bool IsInstalled(const std::string& package_id) const;
// Data management
int64_t GetAppDataSize(const std::string& package_id) const;
bool ClearAppData(const std::string& package_id);
bool ClearAppCache(const std::string& package_id);
bool BackupAppData(const std::string& package_id);
bool RestoreAppData(const std::string& package_id);
// Integration with sandbox
void SetSandboxManager(LuaSandboxManager* manager);
private:
std::string m_data_root;
LuaSandboxManager* m_sandbox_manager = nullptr;
mutable std::mutex m_mutex;
std::map<std::string, InstalledApp> m_installed_apps;
bool VerifyPackage(const std::string& path, const std::string& signature);
bool ExtractPackage(const std::string& path, const std::string& dest);
void LoadInstalledApps();
void SaveInstalledApps();
};
} // namespace mosis
```
### Implementation
```cpp
bool AppManager::Install(const std::string& package_url,
const std::string& signature,
ProgressCallback callback) {
callback({InstallProgress::Stage::Downloading, 0.0f, ""});
// Download package
std::string download_path = m_data_root + "/downloads/" + GenerateUUID();
if (!DownloadFile(package_url, download_path, [&](float p) {
callback({InstallProgress::Stage::Downloading, p, ""});
})) {
callback({InstallProgress::Stage::Failed, 0.0f, "Download failed"});
return false;
}
callback({InstallProgress::Stage::Verifying, 0.0f, ""});
// Verify signature
if (!VerifyPackage(download_path, signature)) {
std::filesystem::remove(download_path);
callback({InstallProgress::Stage::Failed, 0.0f, "Signature verification failed"});
return false;
}
// Extract manifest to get package_id
auto manifest = ExtractManifest(download_path);
if (!manifest) {
std::filesystem::remove(download_path);
callback({InstallProgress::Stage::Failed, 0.0f, "Invalid manifest"});
return false;
}
callback({InstallProgress::Stage::Extracting, 0.0f, ""});
// Check if already installed (update path)
std::string install_path = m_data_root + "/apps/" + manifest->package_id;
if (std::filesystem::exists(install_path + "/package")) {
// Backup existing data
BackupAppData(manifest->package_id);
// Remove old package
std::filesystem::remove_all(install_path + "/package");
}
// Extract package
std::filesystem::create_directories(install_path + "/package");
if (!ExtractPackage(download_path, install_path + "/package")) {
callback({InstallProgress::Stage::Failed, 0.0f, "Extraction failed"});
return false;
}
// Clean up download
std::filesystem::remove(download_path);
callback({InstallProgress::Stage::Registering, 0.0f, ""});
// Create data directories
std::filesystem::create_directories(install_path + "/data");
std::filesystem::create_directories(install_path + "/cache");
std::filesystem::create_directories(install_path + "/db");
// Register app
InstalledApp app{
.package_id = manifest->package_id,
.name = manifest->name,
.version_name = manifest->version,
.version_code = manifest->version_code,
.install_path = install_path,
.permissions = manifest->permissions,
.installed_at = std::chrono::system_clock::now(),
.updated_at = std::chrono::system_clock::now(),
.package_size = std::filesystem::file_size(download_path)
};
{
std::lock_guard<std::mutex> lock(m_mutex);
m_installed_apps[manifest->package_id] = app;
SaveInstalledApps();
}
callback({InstallProgress::Stage::Complete, 1.0f, ""});
return true;
}
bool AppManager::Uninstall(const std::string& package_id, bool keep_data) {
std::lock_guard<std::mutex> lock(m_mutex);
auto it = m_installed_apps.find(package_id);
if (it == m_installed_apps.end()) {
return false;
}
// Stop app if running
if (m_sandbox_manager && m_sandbox_manager->IsAppRunning(package_id)) {
m_sandbox_manager->StopApp(package_id);
}
// Remove files
std::string install_path = it->second.install_path;
std::filesystem::remove_all(install_path + "/package");
std::filesystem::remove_all(install_path + "/cache");
if (!keep_data) {
std::filesystem::remove_all(install_path + "/data");
std::filesystem::remove_all(install_path + "/db");
std::filesystem::remove_all(install_path);
}
// Unregister
m_installed_apps.erase(it);
SaveInstalledApps();
return true;
}
```
---
## Update Service
### Background Update Checker
```cpp
class UpdateService {
public:
UpdateService(AppManager* app_manager, const std::string& api_base);
// Start background checking
void Start(std::chrono::hours interval = std::chrono::hours(24));
void Stop();
// Manual check
std::vector<UpdateInfo> CheckForUpdates();
// Download and install update
bool ApplyUpdate(const std::string& package_id, ProgressCallback callback);
// Settings
void SetAutoUpdate(bool enabled);
void SetWifiOnly(bool wifi_only);
private:
void CheckLoop();
AppManager* m_app_manager;
std::string m_api_base;
std::thread m_check_thread;
std::atomic<bool> m_running{false};
bool m_auto_update = false;
bool m_wifi_only = true;
};
```
### Update Check Flow
```
1. Get list of installed apps
2. Call API: GET /store/apps/updates?packages=com.a,com.b,com.c
3. API returns list of available updates
4. If auto-update enabled and on WiFi:
- Download and install in background
- Notify user of completed updates
5. If manual:
- Show notification with update count
- User opens App Store to review
```
---
## App Store System App
### UI Screens
```
Home
├── Featured apps
├── Categories
├── Search
└── My Apps
├── Installed
├── Updates available
└── Previously installed
App Detail
├── Icon, name, developer
├── Screenshots
├── Description
├── Permissions
├── Reviews (future)
└── [Install] / [Update] / [Open]
Settings
├── Auto-update (WiFi only)
├── Storage usage
└── Clear all caches
```
### RML Structure
```xml
<!-- app_store/main.rml -->
<rml>
<head>
<link type="text/rcss" href="../../ui/theme.rcss"/>
<link type="text/rcss" href="store.rcss"/>
<script src="store.lua"/>
</head>
<body>
<div class="app-bar">
<h1>App Store</h1>
<div class="search-icon" onclick="showSearch()"/>
</div>
<div id="content">
<!-- Dynamic content loaded by Lua -->
</div>
<div class="bottom-nav">
<div class="nav-item active" onclick="showHome()">
<img src="icons/home.tga"/>
<span>Home</span>
</div>
<div class="nav-item" onclick="showCategories()">
<img src="icons/category.tga"/>
<span>Categories</span>
</div>
<div class="nav-item" onclick="showMyApps()">
<img src="icons/apps.tga"/>
<span>My Apps</span>
</div>
</div>
</body>
</rml>
```
### Lua Store Logic
```lua
-- store.lua
local api = require("store_api")
local ui = require("ui")
local state = {
screen = "home",
featured = {},
categories = {},
installed = {},
updates = {}
}
function init()
-- Load installed apps
state.installed = mosis.apps.getInstalled()
-- Fetch featured apps
api.getFeatured(function(apps)
state.featured = apps
render()
end)
-- Check for updates
checkUpdates()
end
function checkUpdates()
local package_ids = {}
for _, app in ipairs(state.installed) do
table.insert(package_ids, app.package_id)
end
api.checkUpdates(package_ids, function(updates)
state.updates = updates
render()
end)
end
function installApp(package_id)
local app = findApp(package_id)
if not app then return end
showProgress(app.name)
mosis.apps.install(app.download_url, app.signature, function(progress)
updateProgress(progress.stage, progress.progress)
if progress.stage == "complete" then
hideProgress()
showToast(app.name .. " installed")
state.installed = mosis.apps.getInstalled()
render()
elseif progress.stage == "failed" then
hideProgress()
showError("Installation failed: " .. progress.error)
end
end)
end
function openApp(package_id)
mosis.apps.launch(package_id)
end
function uninstallApp(package_id)
showConfirm("Uninstall " .. getAppName(package_id) .. "?", function(confirmed)
if confirmed then
mosis.apps.uninstall(package_id)
state.installed = mosis.apps.getInstalled()
render()
end
end)
end
```
---
## Lua API for Apps
### Exposed to System Apps
```lua
-- mosis.apps namespace (system apps only)
-- Get installed apps
local apps = mosis.apps.getInstalled()
-- Returns: [{package_id, name, version_name, version_code, installed_at}]
-- Install from store
mosis.apps.install(url, signature, callback)
-- callback(progress): {stage, progress, error}
-- Uninstall
mosis.apps.uninstall(package_id)
-- Launch app
mosis.apps.launch(package_id)
-- Get app info
local info = mosis.apps.getInfo(package_id)
-- Storage management
local size = mosis.apps.getDataSize(package_id)
mosis.apps.clearCache(package_id)
mosis.apps.clearData(package_id)
```
### Exposed to All Apps
```lua
-- mosis.app namespace (current app only)
-- Get own package info
local info = mosis.app.info()
-- Returns: {package_id, name, version_name, version_code}
-- Check for update
mosis.app.checkUpdate(function(available, new_version)
if available then
showUpdatePrompt(new_version)
end
end)
-- Open store page for self
mosis.app.openStorePage()
```
---
## Permissions for App Management
```lua
-- Required permission to use mosis.apps.*
permissions = {"system.app_management"}
-- Only granted to:
-- - App Store system app
-- - Settings system app
-- - Other OEM system apps
```
---
## Installation Intents
### From Deep Links
```
mosis://store/app/com.developer.myapp
mosis://store/install?url=...&sig=...
```
### From App Store
```lua
-- User taps Install button
installApp("com.developer.myapp")
```
### From ADB (Development)
```bash
adb shell am broadcast -a com.omixlab.mosis.INSTALL_APP \
--es package_path "/sdcard/myapp.mosis"
```
---
## Security
### Package Verification
1. Verify ZIP integrity
2. Verify Ed25519 signature
3. Verify signer is registered developer
4. Verify app not in blocklist
5. Verify permissions are declared
### Installation Sources
| Source | Allowed |
|--------|---------|
| Official store | Always |
| Developer sideload | If enabled in settings |
| Unknown APK | Never (MosisService only) |
### Sandboxing
- All apps run in LuaSandbox
- File access limited to app's data directory
- Network access requires permission
- Hardware access requires permission + user gesture
---
## Deliverables
- [ ] AppManager C++ class
- [ ] UpdateService background checker
- [ ] App Store system app
- [ ] Lua API bindings (mosis.apps, mosis.app)
- [ ] Installation progress UI
- [ ] Uninstall confirmation UI
- [ ] Storage management UI
- [ ] Deep link handling
---
## Open Questions
1. App backup to cloud?
2. Family sharing / multiple devices?
3. Enterprise MDM integration?
4. Sideloading policy?
---
## References
- [Android PackageManager](https://developer.android.com/reference/android/content/pm/PackageManager)
- [iOS App Installation](https://developer.apple.com/documentation/devicemanagement/installing_apps)

609
DEV_PORTAL_M11_CLI.md Normal file
View File

@@ -0,0 +1,609 @@
# Milestone 11: Developer CLI Tool
**Status**: Planning
**Goal**: Command-line tool for app development workflow.
---
## Overview
The CLI tool (`mosis`) streamlines the developer workflow: project creation, building, testing, signing, and publishing.
---
## Commands Overview
```
mosis
├── init Create new app project
├── validate Validate manifest and assets
├── build Create .mosis package
├── sign Sign package with developer key
├── run Run in local designer/emulator
├── test Run automated tests
├── login Authenticate with portal
├── logout Clear authentication
├── publish Upload and submit for review
├── status Check review status
├── keys
│ ├── generate Generate signing keypair
│ ├── list List registered keys
│ └── register Upload public key to portal
└── config
├── get Get config value
└── set Set config value
```
---
## Command Details
### `mosis init`
Create a new app project with boilerplate.
```bash
$ mosis init
? App name: My Calculator
? Package ID: com.myname.calculator
? Description: A simple calculator app
? Author name: John Doe
? Author email: john@example.com
Creating project structure...
✓ Created manifest.json
✓ Created assets/main.rml
✓ Created assets/styles/theme.rcss
✓ Created assets/scripts/app.lua
✓ Created icons/ (placeholder icons)
Project created! Next steps:
cd my-calculator
mosis run # Preview in designer
mosis build # Create package
mosis publish # Submit to store
```
#### Generated Structure
```
my-calculator/
├── manifest.json
├── assets/
│ ├── main.rml
│ ├── styles/
│ │ └── theme.rcss
│ └── scripts/
│ └── app.lua
├── icons/
│ ├── icon-32.png
│ ├── icon-64.png
│ └── icon-128.png
└── .mosisignore # Files to exclude from package
```
---
### `mosis validate`
Validate project without building.
```bash
$ mosis validate
Validating manifest.json...
✓ Required fields present
✓ Package ID format valid
✓ Version format valid
Validating assets...
✓ Entry point exists: assets/main.rml
✓ All RML files valid (3 files)
✓ All RCSS files valid (2 files)
✓ All Lua files valid (4 files)
Validating icons...
✓ icon-32.png (32x32)
✓ icon-64.png (64x64)
✓ icon-128.png (128x128)
Checking permissions...
✓ Permissions declared: storage, network
All validations passed!
```
---
### `mosis build`
Create a .mosis package.
```bash
$ mosis build
Reading manifest.json...
Package: com.myname.calculator v1.0.0 (1)
Collecting files...
✓ manifest.json
✓ assets/main.rml
✓ assets/styles/theme.rcss
✓ assets/scripts/app.lua
✓ icons/icon-32.png
✓ icons/icon-64.png
✓ icons/icon-128.png
Creating package...
✓ Package created: dist/com.myname.calculator-1.0.0.mosis (45.2 KB)
⚠ Package is unsigned. Run 'mosis sign' before publishing.
```
#### Options
```bash
mosis build [options]
Options:
-o, --output <path> Output path (default: dist/)
--no-compress Skip compression
--include-source Include .lua source maps
```
---
### `mosis sign`
Sign a package with developer key.
```bash
$ mosis sign dist/com.myname.calculator-1.0.0.mosis
Using key: ~/.mosis/signing_key.pem
Fingerprint: SHA256:abc123...
Generating file hashes...
Signing MANIFEST.MF...
✓ Package signed: dist/com.myname.calculator-1.0.0.mosis
Signature details:
Algorithm: Ed25519
Key fingerprint: SHA256:abc123...
Files signed: 7
```
#### Options
```bash
mosis sign <package> [options]
Options:
-k, --key <path> Path to private key (default: ~/.mosis/signing_key.pem)
--verify Verify existing signature
```
---
### `mosis run`
Launch in desktop designer for testing.
```bash
$ mosis run
Starting Mosis Designer...
Loading: assets/main.rml
Designer running at http://localhost:8080
Press Ctrl+C to stop
[Hot reload enabled - changes auto-refresh]
```
#### Options
```bash
mosis run [options]
Options:
--entry <file> Override entry point
--port <number> Designer port (default: 8080)
--no-hot-reload Disable hot reload
--device <name> Emulate device (phone, tablet)
```
---
### `mosis login`
Authenticate with developer portal.
```bash
$ mosis login
Opening browser for authentication...
Waiting for authorization...
✓ Logged in as john@example.com
API key stored in ~/.mosis/credentials
```
#### Options
```bash
mosis login [options]
Options:
--api-key <key> Use API key instead of browser auth
--portal <url> Portal URL (default: https://portal.mosis.dev)
```
---
### `mosis publish`
Upload and submit for review.
```bash
$ mosis publish
Checking authentication...
✓ Logged in as john@example.com
Building package...
✓ Package created: dist/com.myname.calculator-1.0.0.mosis
Signing package...
✓ Package signed
Uploading...
████████████████████████████████ 100%
Submitting for review...
✓ Version 1.0.0 submitted
Review status: In Review
Estimated review time: 24 hours
Track status: mosis status
```
#### Options
```bash
mosis publish [options]
Options:
--package <path> Use existing package
--notes <text> Release notes
--notes-file <path> Release notes from file
--draft Upload without submitting for review
```
---
### `mosis status`
Check app/version status.
```bash
$ mosis status
App: My Calculator (com.myname.calculator)
Versions:
v1.0.0 (1) Published Jan 10, 2024 1,234 downloads
v1.1.0 (2) In Review Jan 15, 2024 Submitted 2h ago
Latest review:
Status: In Review
Submitted: Jan 15, 2024 10:30 AM
Estimated completion: Jan 16, 2024
Run 'mosis status --watch' to monitor in real-time.
```
---
### `mosis keys generate`
Generate Ed25519 signing keypair.
```bash
$ mosis keys generate
Generating Ed25519 keypair...
Private key saved to: ~/.mosis/signing_key.pem
Public key saved to: ~/.mosis/signing_key.pub
Fingerprint: SHA256:abc123def456...
⚠ IMPORTANT: Keep your private key secure!
- Never share or commit signing_key.pem
- Back it up securely
- If compromised, revoke immediately
Next step: Register your public key
mosis keys register
```
---
### `mosis keys register`
Upload public key to portal.
```bash
$ mosis keys register
Reading public key from ~/.mosis/signing_key.pub
Fingerprint: SHA256:abc123def456...
? Key name: MacBook Pro 2024
Uploading to portal...
✓ Key registered successfully
Your signing key is now active. Packages signed with this
key will be accepted for review.
```
---
## Configuration
### Config File Location
```
~/.mosis/
├── config.json # CLI configuration
├── credentials # Auth tokens (encrypted)
├── signing_key.pem # Private key
└── signing_key.pub # Public key
```
### Config Options
```json
{
"portal_url": "https://portal.mosis.dev",
"api_url": "https://api.mosis.dev",
"designer_path": "/usr/local/bin/mosis-designer",
"default_author": {
"name": "John Doe",
"email": "john@example.com"
}
}
```
---
## Implementation
### Tech Stack Options
#### Option A: Go
```go
// Using cobra for CLI framework
package main
import (
"github.com/spf13/cobra"
)
func main() {
rootCmd := &cobra.Command{
Use: "mosis",
Short: "Mosis app development CLI",
}
rootCmd.AddCommand(initCmd())
rootCmd.AddCommand(buildCmd())
rootCmd.AddCommand(signCmd())
rootCmd.AddCommand(publishCmd())
// ...
rootCmd.Execute()
}
```
**Pros**: Single binary, fast, cross-platform
**Cons**: More code to write
#### Option B: Node.js (oclif)
```typescript
// Using oclif framework
import { Command } from '@oclif/core'
export default class Build extends Command {
static description = 'Build .mosis package'
async run() {
const manifest = await this.readManifest()
const files = await this.collectFiles()
const package = await this.createPackage(files)
this.log(`✓ Package created: ${package.path}`)
}
}
```
**Pros**: Fast development, npm distribution
**Cons**: Requires Node.js runtime
#### Option C: Rust (clap)
```rust
use clap::{Parser, Subcommand};
#[derive(Parser)]
#[command(name = "mosis")]
struct Cli {
#[command(subcommand)]
command: Commands,
}
#[derive(Subcommand)]
enum Commands {
Init { name: Option<String> },
Build { output: Option<PathBuf> },
Sign { package: PathBuf },
Publish,
}
fn main() {
let cli = Cli::parse();
match cli.command {
Commands::Init { name } => init::run(name),
Commands::Build { output } => build::run(output),
// ...
}
}
```
**Pros**: Single binary, very fast
**Cons**: Slower development
---
## Distribution
### npm (Node.js version)
```bash
npm install -g @mosis/cli
```
### Homebrew (macOS)
```bash
brew tap mosis/tap
brew install mosis
```
### Direct Download
```bash
# Linux/macOS
curl -fsSL https://mosis.dev/install.sh | sh
# Windows
irm https://mosis.dev/install.ps1 | iex
```
### Package Managers
| Platform | Package Manager | Command |
|----------|-----------------|---------|
| macOS | Homebrew | `brew install mosis` |
| Windows | Scoop | `scoop install mosis` |
| Linux | apt (deb) | `apt install mosis` |
| Any | npm | `npm install -g @mosis/cli` |
---
## Error Handling
### User-Friendly Errors
```bash
$ mosis build
Error: manifest.json not found
Are you in a Mosis project directory?
Run 'mosis init' to create a new project.
```
### Verbose Mode
```bash
$ mosis build --verbose
[DEBUG] Reading manifest from ./manifest.json
[DEBUG] Manifest parsed: {id: "com.example.app", ...}
[DEBUG] Collecting files from ./assets
[DEBUG] Found 7 files
[DEBUG] Creating ZIP archive
[DEBUG] Writing to dist/com.example.app-1.0.0.mosis
[DEBUG] Package size: 45234 bytes
✓ Package created
```
---
## CI/CD Integration
### GitHub Actions
```yaml
name: Build and Publish
on:
push:
tags:
- 'v*'
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Mosis CLI
run: npm install -g @mosis/cli
- name: Build and Sign
env:
MOSIS_SIGNING_KEY: ${{ secrets.MOSIS_SIGNING_KEY }}
run: |
echo "$MOSIS_SIGNING_KEY" > signing_key.pem
mosis build
mosis sign dist/*.mosis --key signing_key.pem
- name: Publish
env:
MOSIS_API_KEY: ${{ secrets.MOSIS_API_KEY }}
run: mosis publish --api-key "$MOSIS_API_KEY"
```
---
## Deliverables
- [ ] CLI framework selection
- [ ] `init` command
- [ ] `validate` command
- [ ] `build` command
- [ ] `sign` command
- [ ] `run` command (designer integration)
- [ ] `login/logout` commands
- [ ] `publish` command
- [ ] `status` command
- [ ] `keys` subcommands
- [ ] Configuration management
- [ ] Distribution packages
- [ ] CI/CD examples
---
## Open Questions
1. Should CLI auto-update itself?
2. Offline mode for build/sign?
3. Plugin system for custom commands?
4. IDE integrations (VS Code extension)?
---
## References
- [Cobra CLI Framework](https://cobra.dev/)
- [oclif Framework](https://oclif.io/)
- [Clap for Rust](https://docs.rs/clap/)

617
DEV_PORTAL_M12_DOCS.md Normal file
View File

@@ -0,0 +1,617 @@
# Milestone 12: Documentation Site
**Status**: Planning
**Goal**: Comprehensive documentation for Mosis app developers.
---
## Overview
The documentation site is the primary resource for developers learning to build Mosis apps. Must be clear, searchable, and up-to-date.
---
## Information Architecture
```
docs.mosis.dev/
├── Getting Started
│ ├── Introduction
│ ├── Quick Start (5 min)
│ ├── Your First App
│ └── Project Structure
├── Guides
│ ├── UI Design
│ │ ├── RML Basics
│ │ ├── Styling with RCSS
│ │ ├── Layouts
│ │ └── Components
│ ├── Lua Scripting
│ │ ├── Basics
│ │ ├── Event Handling
│ │ ├── State Management
│ │ └── Async Operations
│ ├── Data & Storage
│ │ ├── Local Storage
│ │ ├── SQLite Database
│ │ └── Files
│ ├── Networking
│ │ ├── HTTP Requests
│ │ └── WebSockets
│ ├── Hardware
│ │ ├── Camera
│ │ ├── Microphone
│ │ ├── Location
│ │ └── Sensors
│ ├── Permissions
│ │ ├── Permission Model
│ │ ├── Requesting Permissions
│ │ └── User Gestures
│ └── Publishing
│ ├── Preparing for Release
│ ├── App Signing
│ └── Store Guidelines
├── API Reference
│ ├── Lua APIs
│ │ ├── mosis.storage
│ │ ├── mosis.db
│ │ ├── mosis.http
│ │ ├── mosis.ws
│ │ ├── mosis.camera
│ │ ├── mosis.microphone
│ │ ├── mosis.location
│ │ ├── mosis.sensors
│ │ └── ...
│ ├── RML Elements
│ ├── RCSS Properties
│ └── Manifest Schema
├── CLI Reference
│ ├── mosis init
│ ├── mosis build
│ ├── mosis sign
│ ├── mosis publish
│ └── ...
├── Best Practices
│ ├── Performance
│ ├── Security
│ ├── UX Guidelines
│ └── Accessibility
├── Troubleshooting
│ ├── Common Errors
│ ├── Debugging Tips
│ └── FAQ
└── Changelog
```
---
## Content Types
### Tutorials (Step-by-step)
```markdown
# Build a Weather App
In this tutorial, you'll build a weather app that:
- Fetches weather data from an API
- Displays current conditions
- Shows a 5-day forecast
- Requests location permission
**Time:** 30 minutes
**Prerequisites:** Completed Quick Start
## Step 1: Create the Project
```bash
mosis init weather-app
cd weather-app
```
## Step 2: Design the UI
Open `assets/main.rml` and add...
```
### Guides (Conceptual)
```markdown
# Understanding Permissions
Mosis uses a permission system to protect user privacy.
Apps must declare permissions in their manifest and
request them at runtime.
## Permission Categories
| Category | Description | Examples |
|----------|-------------|----------|
| Normal | Low risk, auto-granted | storage, network |
| Dangerous | User data, requires prompt | camera, location |
| Signature | System only | app_management |
## When Permissions Are Checked
Permissions are checked when your app calls...
```
### API Reference (Technical)
```markdown
# mosis.http
HTTP client for making network requests.
## Functions
### `mosis.http.get(url, options)`
Make a GET request.
**Parameters:**
- `url` (string): The URL to fetch
- `options` (table, optional):
- `headers` (table): Request headers
- `timeout` (number): Timeout in ms (default: 30000)
**Returns:** Promise that resolves to Response
**Example:**
```lua
local response = mosis.http.get("https://api.example.com/data")
if response.ok then
local data = json.decode(response.body)
print(data.name)
end
```
**Errors:**
- `NETWORK_ERROR`: Network unavailable
- `TIMEOUT`: Request timed out
- `INVALID_URL`: Malformed URL
```
---
## Tech Stack Options
### Option A: Docusaurus
```
Framework: Docusaurus 3
Language: MDX (Markdown + React)
Search: Algolia DocSearch
Deploy: Vercel/Cloudflare Pages
```
| Pros | Cons |
|------|------|
| Versioning built-in | React knowledge needed |
| Great search | Can be heavy |
| Plugin ecosystem | |
| Used by many OSS | |
### Option B: VitePress
```
Framework: VitePress
Language: Markdown + Vue
Search: Built-in local search
Deploy: Any static host
```
| Pros | Cons |
|------|------|
| Very fast | Fewer features |
| Vue-powered | Less ecosystem |
| Simple setup | |
### Option C: Astro Starlight
```
Framework: Astro + Starlight
Language: MDX
Search: Pagefind (local)
Deploy: Any static host
```
| Pros | Cons |
|------|------|
| Very fast | Newer |
| Great defaults | Less customizable |
| Built-in i18n | |
### Option D: MkDocs Material
```
Framework: MkDocs
Language: Markdown
Search: Built-in
Deploy: Any static host
```
| Pros | Cons |
|------|------|
| Simple | Less interactive |
| Great theme | Python-based |
| Fast builds | |
---
## Features Required
### Must Have
- [ ] Full-text search
- [ ] Syntax highlighting
- [ ] Mobile responsive
- [ ] Dark mode
- [ ] Version selector
- [ ] Edit on GitHub links
- [ ] Copy code buttons
- [ ] Table of contents
- [ ] Previous/Next navigation
### Nice to Have
- [ ] API playground
- [ ] Interactive examples
- [ ] Video tutorials
- [ ] Community translations
- [ ] Feedback widget
---
## Code Examples
### Runnable Examples
```html
<!-- Interactive code block -->
<CodePlayground language="lua">
local greeting = "Hello, Mosis!"
print(greeting)
</CodePlayground>
```
### Multi-file Examples
```markdown
:::code-group
```rml [main.rml]
<div id="counter">
<span id="count">0</span>
<button onclick="increment()">+</button>
</div>
```
```lua [app.lua]
local count = 0
function increment()
count = count + 1
document:GetElementById("count").inner_rml = tostring(count)
end
```
```rcss [styles.rcss]
#counter {
display: flex;
gap: 10px;
}
```
:::
```
---
## API Documentation Generation
### From Lua Annotations
```lua
--- Make an HTTP GET request.
--- @param url string The URL to fetch
--- @param options? HttpOptions Request options
--- @return HttpResponse response The response object
--- @example
--- local resp = mosis.http.get("https://api.example.com/data")
--- print(resp.body)
function mosis.http.get(url, options)
-- implementation
end
```
### Generated Output
```markdown
## `mosis.http.get(url, options?)`
Make an HTTP GET request.
### Parameters
| Name | Type | Description |
|------|------|-------------|
| url | string | The URL to fetch |
| options | HttpOptions? | Request options |
### Returns
`HttpResponse` - The response object
### Example
```lua
local resp = mosis.http.get("https://api.example.com/data")
print(resp.body)
```
```
---
## Versioning
### URL Structure
```
docs.mosis.dev/ # Latest stable
docs.mosis.dev/v1.0/ # Version 1.0
docs.mosis.dev/v1.1/ # Version 1.1
docs.mosis.dev/next/ # Development (unreleased)
```
### Version Dropdown
```
┌────────────────────┐
│ Version: 1.1 (latest) ▼ │
├────────────────────┤
│ 1.1 (latest) │
│ 1.0 │
│ next (unreleased) │
└────────────────────┘
```
---
## Search
### Algolia DocSearch (Free for OSS)
```javascript
// docusaurus.config.js
themeConfig: {
algolia: {
appId: 'YOUR_APP_ID',
apiKey: 'YOUR_SEARCH_KEY',
indexName: 'mosis',
},
}
```
### Local Search (Pagefind)
```javascript
// For Astro/VitePress
// Indexes at build time, searches client-side
// No external service needed
```
---
## Internationalization
### Directory Structure
```
docs/
├── en/
│ ├── getting-started/
│ └── guides/
├── es/
│ ├── getting-started/
│ └── guides/
└── zh/
├── getting-started/
└── guides/
```
### Language Switcher
```
┌──────────┐
│ 🌐 EN ▼ │
├──────────┤
│ English │
│ Español │
│ 中文 │
└──────────┘
```
---
## Content Guidelines
### Writing Style
1. **Be concise** - Get to the point quickly
2. **Use active voice** - "The function returns..." not "A value is returned..."
3. **Show, don't tell** - Code examples over explanations
4. **Assume beginner** - Don't assume prior knowledge
5. **Test all examples** - Every code block must work
### Code Style
```lua
-- Good: Clear, commented
local response = mosis.http.get(API_URL)
if response.ok then
local data = json.decode(response.body)
updateUI(data)
else
showError("Failed to load data")
end
-- Bad: Unclear, no error handling
local d = json.decode(mosis.http.get(u).body)
```
### Screenshots
- Use consistent device frame
- Show relevant UI only
- Add callouts for important areas
- Keep file sizes small (WebP)
---
## Deployment
### CI/CD Pipeline
```yaml
# .github/workflows/docs.yml
name: Deploy Docs
on:
push:
branches: [main]
paths:
- 'docs/**'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: 20
- name: Install and Build
run: |
cd docs
npm install
npm run build
- name: Deploy to Cloudflare Pages
uses: cloudflare/pages-action@v1
with:
apiToken: ${{ secrets.CF_API_TOKEN }}
accountId: ${{ secrets.CF_ACCOUNT_ID }}
projectName: mosis-docs
directory: docs/build
```
---
## Analytics
### Track (Privacy-Friendly)
- Page views (which docs are popular)
- Search queries (what are people looking for)
- 404 pages (missing content)
- Time on page (engagement)
### Tools
- Plausible (privacy-focused)
- Simple Analytics
- Cloudflare Analytics (free)
---
## Feedback
### Per-Page Feedback
```
┌────────────────────────────────────┐
│ Was this page helpful? │
│ │
│ [👍 Yes] [👎 No] │
│ │
│ [Edit this page on GitHub] │
└────────────────────────────────────┘
```
### Feedback Collection
```javascript
// Send to analytics or issue tracker
function submitFeedback(page, helpful, comment) {
fetch('/api/feedback', {
method: 'POST',
body: JSON.stringify({ page, helpful, comment })
})
}
```
---
## Deliverables
- [ ] Framework selection
- [ ] Information architecture
- [ ] Getting Started content
- [ ] UI design guides
- [ ] Lua scripting guides
- [ ] API reference (all namespaces)
- [ ] CLI reference
- [ ] Best practices
- [ ] Search integration
- [ ] Version selector
- [ ] Deploy pipeline
- [ ] Feedback system
---
## Content Prioritization
### Phase 1 (Launch)
1. Quick Start
2. Your First App
3. Project Structure
4. RML Basics
5. Lua Basics
6. API Reference (core APIs)
7. CLI Reference
### Phase 2
1. Complete API Reference
2. All hardware guides
3. Best practices
4. Troubleshooting
### Phase 3
1. Advanced guides
2. Video tutorials
3. Translations
4. Community contributions
---
## Open Questions
1. Host docs separately or under main domain?
2. Community wiki/contributions?
3. Video tutorial platform (YouTube, embedded)?
4. Glossary/terminology page?
---
## References
- [Docusaurus](https://docusaurus.io/)
- [VitePress](https://vitepress.dev/)
- [Astro Starlight](https://starlight.astro.build/)
- [Divio Documentation System](https://documentation.divio.com/)

714
DEV_PORTAL_MILESTONES.md Normal file
View File

@@ -0,0 +1,714 @@
# Developer Portal & App Ecosystem Milestones
Planning document for the Mosis app development, distribution, and monitoring ecosystem.
---
## Overview
```
Developer Mosis Platform User Device
─────────────────────────────────────────────────────────────────────────────────
Register Account ──────────────► Developer Portal
│
Create App Project ────────────► App Registry
│
Design UI (RML/RCSS/Lua) ──────► Designer Tool (local)
│
Test Locally ──────────────────► Desktop Designer / Emulator
│
Submit for Review ─────────────► App Store Backend
│
Publish ───────────────────────► CDN / Distribution
│
Install ◄─────────────────── User Request
│
Run in Sandbox ◄──────────── Launch App
│
Telemetry/Crashes ──────────► Analytics Backend
```
---
## Decision Areas
| Area | Options | Status |
|------|---------|--------|
| Web Stack | Node/Express, Go, Rust/Axum, .NET | TBD |
| Database | PostgreSQL, SQLite, MongoDB | TBD |
| Auth | OAuth2/OIDC, API keys, JWT | TBD |
| CDN/Storage | S3, Cloudflare R2, self-hosted | TBD |
| Telemetry | Custom, PostHog, Plausible | TBD |
| Crash Reports | Sentry, custom, Crashlytics | TBD |
| App Format | ZIP, custom package, signed | TBD |
---
## Milestone 1: App Package Format
**Goal**: Define how apps are bundled, signed, and validated.
### Questions to Answer
1. What files comprise an app package?
2. How is the manifest structured?
3. How are apps signed for integrity?
4. How are updates handled (full vs delta)?
5. What metadata is required (name, version, permissions, icons)?
### Considerations
| Approach | Pros | Cons |
|----------|------|------|
| **ZIP archive** | Simple, standard tooling | No built-in signing |
| **Custom format (.mosis)** | Can embed signature, metadata | Custom tooling needed |
| **Signed ZIP** | Best of both, detached signature | Slightly more complex |
### Proposed Structure
```
myapp.mosis/
├── manifest.json # App metadata, permissions, entry point
├── signature.sig # Detached signature (optional for dev)
├── icon.png # App icon (multiple sizes?)
├── assets/
│ ├── main.rml # Entry point UI
│ ├── styles.rcss # Stylesheets
│ └── scripts/
│ └── app.lua # Lua code
└── locales/ # i18n (optional)
├── en.json
└── es.json
```
### Manifest Schema (Draft)
```json
{
"id": "com.developer.appname",
"name": "My App",
"version": "1.0.0",
"version_code": 1,
"entry": "assets/main.rml",
"permissions": ["storage", "network"],
"min_mosis_version": "1.0.0",
"author": {
"name": "Developer Name",
"email": "dev@example.com"
},
"icons": {
"32": "icon-32.png",
"64": "icon-64.png",
"128": "icon-128.png"
}
}
```
### Deliverables
- [ ] Manifest JSON schema specification
- [ ] Package format specification
- [ ] Signing mechanism (key format, algorithm)
- [ ] Package validation tool (CLI)
- [ ] Package creation tool (CLI or integrated in designer)
---
## Milestone 2: Web Stack Selection
**Goal**: Choose backend technologies for the developer portal and app store.
### Options Analysis
#### Option A: Node.js + Express/Fastify
| Aspect | Details |
|--------|---------|
| Language | TypeScript |
| Framework | Express, Fastify, or Hono |
| Pros | Large ecosystem, easy hiring, fast development |
| Cons | Single-threaded, callback complexity |
| Hosting | Vercel, Railway, any VPS |
#### Option B: Go
| Aspect | Details |
|--------|---------|
| Language | Go |
| Framework | Gin, Echo, or Chi |
| Pros | Fast, low memory, single binary deployment |
| Cons | Smaller ecosystem, verbose error handling |
| Hosting | Any VPS, Cloud Run |
#### Option C: Rust + Axum
| Aspect | Details |
|--------|---------|
| Language | Rust |
| Framework | Axum, Actix-web |
| Pros | Maximum performance, memory safety |
| Cons | Steep learning curve, slower development |
| Hosting | Any VPS, Fly.io |
#### Option D: .NET
| Aspect | Details |
|--------|---------|
| Language | C# |
| Framework | ASP.NET Core |
| Pros | Enterprise-ready, great tooling, fast |
| Cons | Heavier runtime, Microsoft ecosystem |
| Hosting | Azure, any VPS |
### Evaluation Criteria
1. **Development speed** - How fast can we iterate?
2. **Performance** - Can it handle scale?
3. **Hosting cost** - Monthly infrastructure cost
4. **Team familiarity** - Learning curve
5. **Ecosystem** - Libraries for auth, storage, etc.
### Deliverables
- [ ] Prototype API in top 2 candidates
- [ ] Benchmark comparison
- [ ] Final selection with rationale
---
## Milestone 3: Database Selection
**Goal**: Choose database for developer accounts, app metadata, analytics.
### Options Analysis
#### Option A: PostgreSQL
| Aspect | Details |
|--------|---------|
| Type | Relational |
| Pros | ACID, JSON support, mature, scalable |
| Cons | Requires management, connection pooling |
| Hosting | Supabase, Neon, RDS, self-hosted |
#### Option B: SQLite + Litestream
| Aspect | Details |
|--------|---------|
| Type | Embedded relational |
| Pros | Zero ops, fast reads, simple backup |
| Cons | Single-writer, limited concurrency |
| Hosting | Embedded in app server |
#### Option C: MongoDB
| Aspect | Details |
|--------|---------|
| Type | Document |
| Pros | Flexible schema, easy start |
| Cons | Less ACID, can get expensive |
| Hosting | Atlas, self-hosted |
### Data Models Preview
```
developers
├── id (UUID)
├── email
├── name
├── api_keys[]
├── created_at
└── verified
apps
├── id (UUID)
├── developer_id (FK)
├── package_id (com.dev.app)
├── name
├── description
├── versions[]
├── status (draft/review/published/suspended)
├── created_at
└── updated_at
app_versions
├── id (UUID)
├── app_id (FK)
├── version_code
├── version_name
├── package_url
├── signature
├── release_notes
├── status
└── published_at
telemetry_events
├── id
├── app_id
├── device_id (anonymized)
├── event_type
├── event_data (JSON)
├── timestamp
└── mosis_version
```
### Deliverables
- [ ] Schema design for all tables
- [ ] Migration strategy
- [ ] Backup/restore plan
- [ ] Final selection with rationale
---
## Milestone 4: Authentication System
**Goal**: Secure developer authentication and app signing.
### Developer Authentication
| Method | Use Case |
|--------|----------|
| OAuth2 (GitHub/Google) | Portal login |
| Email + Password | Alternative login |
| API Keys | CLI tools, CI/CD |
| JWT | Session tokens |
### App Signing
| Approach | Details |
|----------|---------|
| **Developer keypair** | Dev signs with private key, we verify with public |
| **Platform signing** | We sign after review (like iOS) |
| **Both** | Dev signs, we countersign after review |
### Key Management
- Developer generates keypair locally
- Public key uploaded to portal
- Private key never leaves developer machine
- Key rotation supported
### Deliverables
- [ ] OAuth2 integration (GitHub, Google)
- [ ] API key generation and management
- [ ] Developer keypair registration
- [ ] App signature verification
- [ ] JWT token handling
---
## Milestone 5: Developer Portal Frontend
**Goal**: Web interface for developer account and app management.
### Pages Required
| Page | Features |
|------|----------|
| **Landing** | Sign up, sign in, overview |
| **Dashboard** | App list, quick stats |
| **App Details** | Versions, analytics, settings |
| **Create App** | Wizard for new app |
| **Submit Version** | Upload, release notes, submit |
| **API Keys** | Generate, revoke keys |
| **Profile** | Account settings, keys |
| **Docs** | SDK docs, API reference |
### Tech Options
| Option | Pros | Cons |
|--------|------|------|
| Next.js | SSR, React, full-stack | Complexity |
| SvelteKit | Fast, simple, SSR | Smaller ecosystem |
| Astro + React | Static + islands | Newer |
| Plain HTML + htmx | Simple, fast | Limited interactivity |
### Deliverables
- [ ] Framework selection
- [ ] UI component library selection
- [ ] Page wireframes
- [ ] Implementation
---
## Milestone 6: App Store Backend API
**Goal**: REST/GraphQL API for app submission, review, and distribution.
### API Endpoints (Draft)
```
Auth
├── POST /auth/register
├── POST /auth/login
├── POST /auth/logout
├── GET /auth/me
Apps
├── GET /apps # List developer's apps
├── POST /apps # Create new app
├── GET /apps/:id # Get app details
├── PATCH /apps/:id # Update app metadata
├── DELETE /apps/:id # Delete app (if no published versions)
Versions
├── GET /apps/:id/versions # List versions
├── POST /apps/:id/versions # Upload new version
├── GET /apps/:id/versions/:vid # Get version details
├── POST /apps/:id/versions/:vid/submit # Submit for review
├── POST /apps/:id/versions/:vid/publish # Publish (after review)
Public (App Store)
├── GET /store/apps # Browse/search apps
├── GET /store/apps/:id # App store listing
├── GET /store/apps/:id/download # Download latest version
API Keys
├── GET /keys # List API keys
├── POST /keys # Generate new key
├── DELETE /keys/:id # Revoke key
Telemetry (device → server)
├── POST /telemetry/events # Batch event upload
├── POST /telemetry/crash # Crash report
```
### Deliverables
- [ ] OpenAPI specification
- [ ] Rate limiting strategy
- [ ] Authentication middleware
- [ ] Implementation
---
## Milestone 7: CDN & Storage
**Goal**: Scalable storage for app packages and assets.
### Requirements
1. Store app packages (.mosis files)
2. Serve downloads globally with low latency
3. Handle icons and screenshots
4. Version retention policy
5. Bandwidth cost management
### Options
| Option | Pros | Cons |
|--------|------|------|
| **Cloudflare R2** | No egress fees, global | Newer service |
| **AWS S3 + CloudFront** | Mature, reliable | Egress costs |
| **Backblaze B2 + Cloudflare** | Cheap storage, free egress via CF | More setup |
| **Self-hosted MinIO** | Full control | Ops burden |
### Storage Structure
```
/packages/
/{app_id}/
/{version_code}/
package.mosis
signature.sig
/assets/
/{app_id}/
icon-32.png
icon-64.png
icon-128.png
screenshots/
1.png
2.png
```
### Deliverables
- [ ] Storage provider selection
- [ ] CDN configuration
- [ ] Upload flow (presigned URLs vs direct)
- [ ] Download URL generation
- [ ] Retention/cleanup policy
---
## Milestone 8: Telemetry System
**Goal**: Collect app usage analytics and crash reports.
### Event Types
| Category | Events |
|----------|--------|
| **Lifecycle** | app_start, app_stop, app_crash |
| **Performance** | frame_time, memory_usage, lua_errors |
| **Usage** | screen_view, button_click (opt-in) |
| **System** | mosis_version, device_info |
### Privacy Considerations
1. **No PII by default** - Device ID is hashed, no user data
2. **Opt-in for detailed analytics** - User consent required
3. **Data retention** - Auto-delete after X days
4. **GDPR compliance** - Export/delete on request
5. **Aggregation** - Store aggregates, drop raw after processing
### Options
| Option | Pros | Cons |
|--------|------|------|
| **Custom** | Full control, no vendor lock | Build everything |
| **PostHog** | Self-hostable, feature-rich | Can be heavy |
| **Plausible** | Privacy-focused, simple | Limited features |
| **Segment + warehouse** | Flexible routing | Complex, costly |
### Crash Report Schema
```json
{
"app_id": "com.dev.app",
"app_version": "1.0.0",
"mosis_version": "1.0.0",
"device_id": "hashed",
"timestamp": "2024-01-15T10:30:00Z",
"crash_type": "lua_error",
"message": "attempt to index nil value",
"stack_trace": "...",
"context": {
"screen": "main.rml",
"memory_mb": 45,
"uptime_seconds": 120
}
}
```
### Deliverables
- [ ] Event schema specification
- [ ] Collection endpoint
- [ ] Storage strategy (time-series DB?)
- [ ] Dashboard for developers
- [ ] Privacy controls
---
## Milestone 9: App Review System
**Goal**: Automated and manual review process for app submissions.
### Automated Checks
| Check | Description |
|-------|-------------|
| **Manifest validation** | Required fields, valid permissions |
| **Package integrity** | Signature verification |
| **Static analysis** | Dangerous Lua patterns |
| **Asset validation** | Icons present, correct sizes |
| **Size limits** | Package under max size |
| **Duplicate detection** | Same app ID collision |
### Manual Review (Optional)
- Flag for manual review based on:
- New developer (first app)
- Dangerous permissions requested
- Automated check warnings
- User reports
### Review States
```
draft → submitted → in_review → approved → published
↘ rejected (with feedback)
```
### Deliverables
- [ ] Automated validation pipeline
- [ ] Review queue UI
- [ ] Rejection feedback system
- [ ] Appeal process
---
## Milestone 10: Device-Side App Management
**Goal**: Install, update, and manage apps on Mosis devices.
### Components
| Component | Location | Purpose |
|-----------|----------|---------|
| App Manager | MosisService | Install/uninstall/update apps |
| App Store Client | System app | Browse, search, install UI |
| Update Checker | Background service | Check for updates |
### Installation Flow
```
1. User taps "Install" in App Store
2. Download package from CDN
3. Verify signature
4. Extract to app directory
5. Register with LuaSandboxManager
6. Add to home screen
```
### Update Flow
```
1. Background check for updates (daily?)
2. Notify user of available updates
3. Download new version
4. Verify signature
5. Replace app files (atomic swap)
6. Restart app if running
```
### Storage Layout
```
/data/mosis/
/apps/
/com.dev.app/
/package/ # Extracted app files
/data/ # App data (VirtualFS)
/cache/ # App cache
/db/ # SQLite databases
```
### Deliverables
- [ ] AppManager class in MosisService
- [ ] App Store system app
- [ ] Update checking service
- [ ] Uninstall with data cleanup
---
## Milestone 11: Developer CLI Tool
**Goal**: Command-line tool for app development workflow.
### Commands
```bash
# Project management
mosis init # Create new app project
mosis validate # Validate manifest and assets
# Packaging
mosis build # Create .mosis package
mosis sign # Sign package with developer key
# Testing
mosis run # Run in local designer/emulator
mosis test # Run automated tests
# Publishing
mosis login # Authenticate with portal
mosis publish # Upload and submit for review
mosis status # Check review status
# Keys
mosis keys generate # Generate signing keypair
mosis keys register # Upload public key to portal
```
### Implementation Options
| Option | Pros | Cons |
|--------|------|------|
| Node.js (oclif) | Easy to build, npm distribution | Requires Node |
| Go | Single binary, fast | Slower development |
| Rust (clap) | Single binary, fast | Slower development |
### Deliverables
- [ ] CLI framework selection
- [ ] Core commands implementation
- [ ] Distribution (npm, homebrew, direct download)
- [ ] Documentation
---
## Milestone 12: Documentation Site
**Goal**: Comprehensive docs for developers.
### Sections
| Section | Content |
|---------|---------|
| **Getting Started** | Quick start, first app tutorial |
| **Guides** | UI design, Lua scripting, permissions |
| **API Reference** | All Lua APIs, manifest schema |
| **CLI Reference** | All commands and options |
| **Best Practices** | Performance, security, UX |
| **Troubleshooting** | Common issues, FAQ |
### Tech Options
| Option | Pros | Cons |
|--------|------|------|
| Docusaurus | React-based, versioning | Heavy |
| VitePress | Vue-based, fast | Less features |
| Astro Starlight | Fast, modern | Newer |
| MkDocs | Python, simple | Less customizable |
### Deliverables
- [ ] Framework selection
- [ ] Information architecture
- [ ] Content writing
- [ ] API docs generation from code
- [ ] Search integration
---
## Summary
| Phase | Milestones | Description |
|-------|------------|-------------|
| **Foundation** | 1-4 | Package format, web stack, database, auth |
| **Portal** | 5-6 | Developer portal frontend and API |
| **Distribution** | 7, 10 | CDN/storage, device-side app management |
| **Quality** | 8-9 | Telemetry, crash reports, review system |
| **Tooling** | 11-12 | CLI tool, documentation |
### Recommended Order
1. **Milestone 1** - Package format (needed by everything)
2. **Milestone 2** - Web stack selection
3. **Milestone 3** - Database selection
4. **Milestone 4** - Authentication
5. **Milestone 6** - Backend API
6. **Milestone 5** - Portal frontend
7. **Milestone 7** - CDN/storage
8. **Milestone 10** - Device-side management
9. **Milestone 11** - CLI tool
10. **Milestone 9** - Review system
11. **Milestone 8** - Telemetry
12. **Milestone 12** - Documentation
---
## Open Questions
1. **Monetization model?** - Free only, paid apps, subscriptions?
2. **Enterprise/self-hosted?** - Can companies run private app stores?
3. **App categories?** - Predefined or free-form tags?
4. **Rating/reviews?** - User reviews for apps?
5. **Analytics dashboard?** - What metrics do developers see?
6. **Localization?** - Multi-language portal and apps?
7. **Beta testing?** - TestFlight-like distribution?
8. **Team accounts?** - Multiple developers per app?
---
## Next Steps
Begin with Milestone 1 (App Package Format) to establish the foundation, then proceed with technology selections in Milestones 2-4 before building the portal.