diff --git a/DEV_PORTAL_M01_APP_PACKAGE.md b/DEV_PORTAL_M01_APP_PACKAGE.md new file mode 100644 index 0000000..5fd742e --- /dev/null +++ b/DEV_PORTAL_M01_APP_PACKAGE.md @@ -0,0 +1,372 @@ +# Milestone 1: App Package Format + +**Status**: Planning +**Goal**: Define how apps are bundled, signed, and validated. + +--- + +## Overview + +The app package format is the foundation of the entire ecosystem. Every tool (CLI, portal, device) needs to understand this format. + +--- + +## Questions to Answer + +1. What files comprise an app package? +2. How is the manifest structured? +3. How are apps signed for integrity? +4. How are updates handled (full vs delta)? +5. What metadata is required (name, version, permissions, icons)? + +--- + +## Package Format Options + +### Option A: ZIP Archive + +``` +myapp.zip +├── manifest.json +├── icon.png +└── assets/ + ├── main.rml + ├── styles.rcss + └── app.lua +``` + +| Pros | Cons | +|------|------| +| Simple, standard tooling | No built-in signing | +| Easy to inspect | No metadata in filename | +| Wide compatibility | Need separate signature file | + +### Option B: Custom Format (.mosis) + +``` +myapp.mosis (binary format) +┌─────────────────────────┐ +│ Magic bytes (4) │ +│ Version (2) │ +│ Manifest length (4) │ +│ Manifest JSON │ +│ Signature (256) │ +│ Compressed payload │ +└─────────────────────────┘ +``` + +| Pros | Cons | +|------|------| +| Signature built-in | Custom tooling needed | +| Efficient metadata access | Harder to inspect | +| Single file | More complex implementation | + +### Option C: Signed ZIP (Recommended) + +``` +myapp.mosis/ (actually a ZIP) +├── manifest.json +├── META-INF/ +│ ├── MANIFEST.MF # File hashes +│ └── SIGNATURE.SF # Signed manifest +├── icon.png +└── assets/ + └── ... +``` + +| Pros | Cons | +|------|------| +| Standard ZIP tooling | Slightly more complex | +| JAR/APK-style signing | | +| Easy inspection | | +| Proven approach | | + +--- + +## Proposed Structure + +``` +com.developer.appname-1.0.0.mosis +├── manifest.json # App metadata +├── META-INF/ +│ ├── CERT.PEM # Developer certificate +│ ├── CERT.SIG # Signature of MANIFEST.MF +│ └── MANIFEST.MF # SHA-256 hashes of all files +├── icons/ +│ ├── icon-32.png +│ ├── icon-64.png +│ └── icon-128.png +├── assets/ +│ ├── main.rml # Entry point +│ ├── screens/ +│ │ ├── home.rml +│ │ └── settings.rml +│ ├── styles/ +│ │ └── theme.rcss +│ └── scripts/ +│ ├── main.lua +│ └── utils.lua +└── locales/ # Optional i18n + ├── en.json + └── es.json +``` + +--- + +## Manifest Schema + +### Required Fields + +```json +{ + "$schema": "https://mosis.dev/schemas/manifest-v1.json", + "id": "com.developer.appname", + "name": "My App", + "version": "1.0.0", + "version_code": 1, + "entry": "assets/main.rml", + "min_mosis_version": "1.0.0" +} +``` + +### Full Schema + +```json +{ + "$schema": "https://mosis.dev/schemas/manifest-v1.json", + + "id": "com.developer.appname", + "name": "My App", + "version": "1.0.0", + "version_code": 1, + + "description": "A short description of the app", + "entry": "assets/main.rml", + + "author": { + "name": "Developer Name", + "email": "dev@example.com", + "url": "https://developer.com" + }, + + "permissions": [ + "storage", + "network", + "camera" + ], + + "icons": { + "32": "icons/icon-32.png", + "64": "icons/icon-64.png", + "128": "icons/icon-128.png" + }, + + "min_mosis_version": "1.0.0", + "target_mosis_version": "1.2.0", + + "category": "utilities", + "tags": ["productivity", "tools"], + + "orientation": "portrait", + "background_color": "#FFFFFF", + + "locales": ["en", "es", "fr"], + "default_locale": "en" +} +``` + +### Field Definitions + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `id` | string | Yes | Unique package identifier (reverse domain) | +| `name` | string | Yes | Display name (max 30 chars) | +| `version` | string | Yes | Semantic version (X.Y.Z) | +| `version_code` | integer | Yes | Incremental build number | +| `entry` | string | Yes | Path to entry RML file | +| `min_mosis_version` | string | Yes | Minimum Mosis version required | +| `description` | string | No | Short description (max 80 chars) | +| `author` | object | No | Author information | +| `permissions` | array | No | Required permissions | +| `icons` | object | No | Icon paths by size | +| `category` | string | No | App store category | +| `tags` | array | No | Searchable tags | +| `orientation` | string | No | portrait, landscape, any | +| `background_color` | string | No | Hex color for loading | +| `locales` | array | No | Supported locales | + +--- + +## Signing Mechanism + +### Algorithm + +- **Key type**: Ed25519 (fast, secure, small signatures) +- **Hash**: SHA-256 for file manifests +- **Format**: PEM for keys, base64 for signatures + +### MANIFEST.MF Format + +``` +Manifest-Version: 1.0 +Created-By: mosis-cli 1.0.0 + +Name: manifest.json +SHA-256-Digest: base64encodedHash== + +Name: assets/main.rml +SHA-256-Digest: base64encodedHash== + +Name: assets/scripts/main.lua +SHA-256-Digest: base64encodedHash== +``` + +### Signing Flow + +``` +1. Generate MANIFEST.MF with SHA-256 of each file +2. Sign MANIFEST.MF with developer's Ed25519 private key +3. Store signature in META-INF/CERT.SIG +4. Include developer's public key in META-INF/CERT.PEM +``` + +### Verification Flow + +``` +1. Extract META-INF/MANIFEST.MF +2. Verify signature using CERT.PEM +3. Verify CERT.PEM is registered with developer account +4. Verify each file hash matches MANIFEST.MF +``` + +--- + +## Size Limits + +| Limit | Value | Rationale | +|-------|-------|-----------| +| Max package size | 50 MB | Reasonable for mobile | +| Max individual file | 10 MB | Prevent abuse | +| Max files count | 1000 | Prevent zip bombs | +| Max path length | 256 chars | Filesystem compat | +| Max manifest size | 64 KB | Prevent abuse | + +--- + +## Validation Rules + +### Manifest Validation + +- [ ] Valid JSON +- [ ] All required fields present +- [ ] `id` matches reverse domain pattern: `^[a-z][a-z0-9]*(\.[a-z][a-z0-9]*)+$` +- [ ] `version` matches semver pattern +- [ ] `version_code` is positive integer +- [ ] `entry` file exists in package +- [ ] All `permissions` are valid permission names +- [ ] All `icons` paths exist and are valid images + +### Package Validation + +- [ ] Valid ZIP format +- [ ] manifest.json at root +- [ ] No path traversal (../) +- [ ] No absolute paths +- [ ] No symlinks +- [ ] Under size limits +- [ ] No duplicate files +- [ ] Valid file extensions only + +### Signature Validation + +- [ ] MANIFEST.MF present +- [ ] CERT.SIG present +- [ ] CERT.PEM present +- [ ] Signature valid for MANIFEST.MF +- [ ] All file hashes match +- [ ] Certificate registered with developer account (for store) + +--- + +## File Extension Rules + +### Allowed Extensions + +| Category | Extensions | +|----------|------------| +| UI | .rml | +| Styles | .rcss | +| Scripts | .lua | +| Images | .png, .jpg, .jpeg, .tga, .webp | +| Fonts | .ttf, .otf | +| Data | .json | +| Localization | .json (in locales/) | +| Audio | .ogg, .wav, .mp3 | + +### Forbidden + +- Executables: .exe, .dll, .so, .dylib +- Scripts: .sh, .bat, .ps1, .py, .js +- Archives: .zip, .tar, .gz (nested) + +--- + +## Update Handling + +### Full Update + +- Download complete new package +- Verify signature +- Atomic replacement of app directory +- Preserve user data in separate location + +### Delta Updates (Future) + +- bsdiff-style patches +- Reduces bandwidth for minor updates +- More complex implementation +- Consider for v2 + +--- + +## Deliverables + +- [ ] JSON Schema for manifest validation +- [ ] Package format specification document +- [ ] Reference implementation: package creator (Go/Rust) +- [ ] Reference implementation: package validator +- [ ] Reference implementation: signature tools +- [ ] Integration with mosis-cli + +--- + +## Test Cases + +| Test | Description | +|------|-------------| +| ValidPackage | Accept well-formed package | +| InvalidManifest | Reject malformed JSON | +| MissingRequired | Reject missing required fields | +| BadSignature | Reject invalid signature | +| TamperedFile | Reject if file hash mismatch | +| PathTraversal | Reject ../ in paths | +| OversizePackage | Reject over size limit | +| BadExtension | Reject forbidden file types | +| DuplicateFiles | Reject duplicate entries | + +--- + +## Open Questions + +1. Should we support multiple entry points (e.g., widget vs full app)? +2. Should icons be required or have defaults? +3. Delta updates in v1 or defer to v2? +4. Support for app bundles (multiple apps in one package)? + +--- + +## References + +- [Android APK format](https://developer.android.com/guide/components/fundamentals) +- [JAR signing](https://docs.oracle.com/javase/tutorial/deployment/jar/signing.html) +- [Ed25519 specification](https://ed25519.cr.yp.to/) diff --git a/DEV_PORTAL_M02_WEB_STACK.md b/DEV_PORTAL_M02_WEB_STACK.md new file mode 100644 index 0000000..3404392 --- /dev/null +++ b/DEV_PORTAL_M02_WEB_STACK.md @@ -0,0 +1,333 @@ +# Milestone 2: Web Stack Selection + +**Status**: Planning +**Goal**: Choose backend technologies for the developer portal and app store API. + +--- + +## Overview + +The web stack powers the developer portal, app store API, and telemetry ingestion. This decision affects development speed, hosting costs, and long-term maintenance. + +--- + +## Requirements + +### Functional + +- REST API for developer portal +- File upload handling (app packages) +- Authentication (OAuth2, API keys) +- Database integration +- Background jobs (review queue, notifications) +- WebSocket support (optional, for real-time updates) + +### Non-Functional + +- Handle 1000+ developers initially +- Scale to 10,000+ apps +- 99.9% uptime target +- < 200ms API response time (p95) +- Secure by default + +--- + +## Options Analysis + +### Option A: Node.js + TypeScript + +#### Stack +``` +Runtime: Node.js 20 LTS +Language: TypeScript +Framework: Fastify or Hono +ORM: Prisma or Drizzle +Validation: Zod +Auth: Lucia or custom JWT +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Fast development | Large ecosystem, familiar syntax | +| Type safety | TypeScript catches errors early | +| Easy hiring | Many JS/TS developers | +| Ecosystem | npm has libraries for everything | +| Hosting | Vercel, Railway, Render, any VPS | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Single-threaded | Need clustering for CPU tasks | +| Memory usage | Higher than compiled languages | +| Callback complexity | Async can get messy | + +#### Example Structure +``` +src/ +├── index.ts +├── routes/ +│ ├── auth.ts +│ ├── apps.ts +│ └── telemetry.ts +├── services/ +│ ├── auth.service.ts +│ └── storage.service.ts +├── db/ +│ ├── schema.ts +│ └── client.ts +└── utils/ +``` + +#### Hosting Cost Estimate +| Service | Cost/month | +|---------|------------| +| Railway (starter) | $5-20 | +| Vercel Pro | $20 | +| VPS (4GB) | $20-40 | + +--- + +### Option B: Go + +#### Stack +``` +Language: Go 1.22+ +Framework: Chi, Echo, or Gin +ORM: sqlc or GORM +Validation: go-playground/validator +Auth: Custom JWT +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Performance | Fast, low memory | +| Single binary | Easy deployment | +| Concurrency | Goroutines handle load well | +| Standard library | HTTP, JSON, crypto built-in | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Verbose | Error handling boilerplate | +| Smaller ecosystem | Fewer ready-made solutions | +| Learning curve | Different paradigm | + +#### Example Structure +``` +cmd/ +├── server/ +│ └── main.go +internal/ +├── api/ +│ ├── handlers/ +│ ├── middleware/ +│ └── routes.go +├── domain/ +│ ├── app.go +│ └── developer.go +├── repository/ +│ └── postgres/ +└── service/ +``` + +#### Hosting Cost Estimate +| Service | Cost/month | +|---------|------------| +| Fly.io (shared) | $5-15 | +| Cloud Run | Pay per use | +| VPS (2GB) | $10-20 | + +--- + +### Option C: Rust + Axum + +#### Stack +``` +Language: Rust +Framework: Axum +ORM: sqlx or SeaORM +Validation: validator crate +Auth: Custom JWT +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Maximum performance | Fastest option | +| Memory safety | No runtime errors | +| Single binary | Easy deployment | +| Long-term reliability | Compiler catches bugs | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Steep learning curve | Borrow checker takes time | +| Slower development | More code to write | +| Smaller ecosystem | Fewer web libraries | +| Compile times | Can be slow | + +#### Example Structure +``` +src/ +├── main.rs +├── api/ +│ ├── mod.rs +│ ├── handlers/ +│ └── middleware/ +├── domain/ +├── repository/ +└── service/ +``` + +#### Hosting Cost Estimate +| Service | Cost/month | +|---------|------------| +| Fly.io | $5-10 | +| Shuttle | Free tier available | +| VPS (1GB) | $5-10 | + +--- + +### Option D: .NET (ASP.NET Core) + +#### Stack +``` +Language: C# +Framework: ASP.NET Core 8 +ORM: Entity Framework Core +Validation: FluentValidation +Auth: ASP.NET Identity +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Enterprise-ready | Battle-tested at scale | +| Great tooling | Visual Studio, debugging | +| Performance | Very fast (Kestrel) | +| Full-featured | Auth, validation, DI built-in | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Heavier runtime | Larger container images | +| Microsoft ecosystem | May feel locked in | +| Verbose | More boilerplate | + +#### Hosting Cost Estimate +| Service | Cost/month | +|---------|------------| +| Azure App Service | $15-50 | +| VPS (4GB) | $20-40 | + +--- + +## Evaluation Matrix + +| Criteria | Weight | Node.js | Go | Rust | .NET | +|----------|--------|---------|-----|------|------| +| Dev speed | 25% | 9 | 7 | 5 | 7 | +| Performance | 20% | 6 | 9 | 10 | 8 | +| Hosting cost | 15% | 7 | 9 | 9 | 6 | +| Ecosystem | 15% | 9 | 7 | 6 | 8 | +| Team familiarity | 15% | ? | ? | ? | ? | +| Long-term maintenance | 10% | 7 | 8 | 9 | 8 | +| **Weighted Score** | | **7.4** | **7.8** | **7.2** | **7.3** | + +*Team familiarity needs to be filled in based on actual experience* + +--- + +## Framework Comparison + +### Node.js Frameworks + +| Framework | Req/sec | Features | Learning Curve | +|-----------|---------|----------|----------------| +| Express | 15k | Minimal, flexible | Easy | +| Fastify | 45k | Fast, schema validation | Medium | +| Hono | 50k | Ultra-light, edge-ready | Easy | +| NestJS | 20k | Full-featured, Angular-like | Steep | + +### Go Frameworks + +| Framework | Req/sec | Features | Learning Curve | +|-----------|---------|----------|----------------| +| net/http | 60k | Standard library | Medium | +| Chi | 55k | Lightweight router | Easy | +| Gin | 50k | Popular, middleware | Easy | +| Echo | 55k | Similar to Gin | Easy | + +--- + +## Prototype Plan + +### Phase 1: Build minimal API in top 2 choices + +Endpoints to implement: +``` +POST /auth/register +POST /auth/login +GET /apps +POST /apps +POST /apps/:id/versions +``` + +### Phase 2: Benchmark + +- Requests per second +- Memory usage under load +- Development time tracking +- Code complexity comparison + +### Phase 3: Decision + +Based on: +- Benchmark results +- Developer experience during prototype +- Hosting cost analysis + +--- + +## Recommendation + +**Primary: Go with Chi/Echo** +- Best balance of performance and simplicity +- Low hosting costs +- Single binary deployment +- Good enough ecosystem for our needs + +**Fallback: Node.js with Hono/Fastify** +- If Go feels too slow to develop in +- Larger ecosystem for edge cases +- More developers familiar with it + +--- + +## Deliverables + +- [ ] Prototype API in Go +- [ ] Prototype API in Node.js (if needed) +- [ ] Benchmark comparison document +- [ ] Final selection with rationale +- [ ] Project structure template +- [ ] Development environment setup guide + +--- + +## Open Questions + +1. Do we need GraphQL or is REST sufficient? +2. Do we need real-time features (WebSocket)? +3. What's the team's current language experience? +4. Any preference for specific cloud providers? + +--- + +## References + +- [TechEmpower Benchmarks](https://www.techempower.com/benchmarks/) +- [Go vs Node.js comparison](https://blog.logrocket.com/node-js-vs-golang/) +- [Fastify benchmarks](https://fastify.dev/benchmarks/) diff --git a/DEV_PORTAL_M03_DATABASE.md b/DEV_PORTAL_M03_DATABASE.md new file mode 100644 index 0000000..c441c0b --- /dev/null +++ b/DEV_PORTAL_M03_DATABASE.md @@ -0,0 +1,418 @@ +# Milestone 3: Database Selection + +**Status**: Planning +**Goal**: Choose database for developer accounts, app metadata, and analytics. + +--- + +## Overview + +The database stores all persistent data: developer accounts, app metadata, versions, telemetry events, and audit logs. + +--- + +## Requirements + +### Data Characteristics + +| Data Type | Volume | Access Pattern | Consistency | +|-----------|--------|----------------|-------------| +| Developers | 10K rows | Read-heavy, low write | Strong | +| Apps | 100K rows | Read-heavy | Strong | +| Versions | 500K rows | Read-heavy | Strong | +| API Keys | 50K rows | Read-heavy | Strong | +| Telemetry | 100M+ rows | Write-heavy, append | Eventual OK | +| Audit Logs | 10M+ rows | Write-heavy, append | Eventual OK | + +### Query Patterns + +- Get developer by email +- List apps by developer +- Get app with latest version +- Search apps by name/tags +- Aggregate telemetry by app/day +- Time-range queries on events + +--- + +## Options Analysis + +### Option A: PostgreSQL + +#### Characteristics +``` +Type: Relational (SQL) +ACID: Full +JSON: Native JSONB support +Full-text: Built-in tsvector +Scaling: Vertical + read replicas +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Battle-tested | Decades of reliability | +| ACID compliance | Strong consistency | +| JSON support | JSONB for flexible data | +| Full-text search | No separate search engine needed | +| Extensions | PostGIS, pg_trgm, etc. | +| Tooling | pgAdmin, great ORMs | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Ops overhead | Need connection pooling | +| Scaling writes | Vertical scaling limits | +| Time-series | Not optimized for telemetry | + +#### Hosting Options +| Provider | Free Tier | Paid | +|----------|-----------|------| +| Supabase | 500MB | $25/mo | +| Neon | 512MB | $19/mo | +| Railway | 1GB | $5/mo | +| AWS RDS | - | $15/mo+ | +| Self-hosted | - | VPS cost | + +--- + +### Option B: SQLite + Litestream + +#### Characteristics +``` +Type: Embedded relational +ACID: Full +Scaling: Single writer +Backup: Litestream to S3 +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Zero ops | No separate DB server | +| Fast reads | In-process, no network | +| Simple backup | Litestream handles replication | +| Low cost | Just storage costs | +| Portable | Easy local development | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Single writer | Limits write concurrency | +| No horizontal scale | One server only | +| Limited features | No full-text (without FTS5) | + +#### Cost Estimate +| Component | Cost/month | +|-----------|------------| +| S3 storage (10GB) | $0.25 | +| Compute | Included in app server | + +--- + +### Option C: PostgreSQL + TimescaleDB + +#### Characteristics +``` +Type: Time-series extension +Base: PostgreSQL +Scaling: Automatic partitioning +Compression: Native +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Best of both | Relational + time-series | +| Auto-partition | Handles telemetry scale | +| Compression | 90%+ compression ratio | +| Continuous aggregates | Pre-computed rollups | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Complexity | More to manage | +| Cost | Higher than plain Postgres | +| Learning curve | New concepts | + +--- + +### Option D: Hybrid Approach + +``` +PostgreSQL → Developers, Apps, Versions, API Keys +ClickHouse/QuestDB → Telemetry, Analytics +Redis → Caching, Sessions +``` + +#### Pros +| Advantage | Details | +|-----------|---------| +| Right tool for job | Optimized for each use case | +| Scale independently | Telemetry won't affect main DB | +| Performance | Best possible for each workload | + +#### Cons +| Disadvantage | Details | +|--------------|---------| +| Complexity | Multiple systems to manage | +| Cost | More infrastructure | +| Consistency | Cross-DB transactions hard | + +--- + +## Schema Design + +### Core Tables + +```sql +-- Developers +CREATE TABLE developers ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + email VARCHAR(255) UNIQUE NOT NULL, + name VARCHAR(100) NOT NULL, + password_hash VARCHAR(255), + oauth_provider VARCHAR(50), + oauth_id VARCHAR(255), + verified BOOLEAN DEFAULT FALSE, + created_at TIMESTAMPTZ DEFAULT NOW(), + updated_at TIMESTAMPTZ DEFAULT NOW() +); + +-- API Keys +CREATE TABLE api_keys ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + developer_id UUID REFERENCES developers(id) ON DELETE CASCADE, + name VARCHAR(100) NOT NULL, + key_hash VARCHAR(255) NOT NULL, + key_prefix VARCHAR(10) NOT NULL, -- For display: "mk_abc..." + permissions JSONB DEFAULT '[]', + last_used_at TIMESTAMPTZ, + expires_at TIMESTAMPTZ, + created_at TIMESTAMPTZ DEFAULT NOW() +); + +-- Apps +CREATE TABLE apps ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + developer_id UUID REFERENCES developers(id) ON DELETE CASCADE, + package_id VARCHAR(255) UNIQUE NOT NULL, -- com.dev.app + name VARCHAR(100) NOT NULL, + description TEXT, + category VARCHAR(50), + tags VARCHAR(50)[] DEFAULT '{}', + status VARCHAR(20) DEFAULT 'draft', -- draft, published, suspended + created_at TIMESTAMPTZ DEFAULT NOW(), + updated_at TIMESTAMPTZ DEFAULT NOW() +); + +-- App Versions +CREATE TABLE app_versions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + app_id UUID REFERENCES apps(id) ON DELETE CASCADE, + version_code INTEGER NOT NULL, + version_name VARCHAR(20) NOT NULL, + package_url TEXT NOT NULL, + package_size BIGINT NOT NULL, + signature VARCHAR(512) NOT NULL, + permissions JSONB DEFAULT '[]', + min_mosis_version VARCHAR(20), + release_notes TEXT, + status VARCHAR(20) DEFAULT 'draft', -- draft, review, approved, published, rejected + review_notes TEXT, + published_at TIMESTAMPTZ, + created_at TIMESTAMPTZ DEFAULT NOW(), + UNIQUE(app_id, version_code) +); + +-- Developer Signing Keys +CREATE TABLE signing_keys ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + developer_id UUID REFERENCES developers(id) ON DELETE CASCADE, + name VARCHAR(100) NOT NULL, + public_key TEXT NOT NULL, + fingerprint VARCHAR(64) NOT NULL, + is_active BOOLEAN DEFAULT TRUE, + created_at TIMESTAMPTZ DEFAULT NOW() +); +``` + +### Telemetry Tables (if using PostgreSQL) + +```sql +-- Telemetry Events (consider partitioning by time) +CREATE TABLE telemetry_events ( + id BIGSERIAL, + app_id UUID NOT NULL, + device_id VARCHAR(64) NOT NULL, -- Hashed + event_type VARCHAR(50) NOT NULL, + event_data JSONB, + mosis_version VARCHAR(20), + timestamp TIMESTAMPTZ NOT NULL, + PRIMARY KEY (timestamp, id) +) PARTITION BY RANGE (timestamp); + +-- Create monthly partitions +CREATE TABLE telemetry_events_2024_01 PARTITION OF telemetry_events + FOR VALUES FROM ('2024-01-01') TO ('2024-02-01'); + +-- Crash Reports +CREATE TABLE crash_reports ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + app_id UUID NOT NULL, + app_version VARCHAR(20) NOT NULL, + device_id VARCHAR(64) NOT NULL, + crash_type VARCHAR(50) NOT NULL, + message TEXT, + stack_trace TEXT, + context JSONB, + mosis_version VARCHAR(20), + timestamp TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ DEFAULT NOW() +); + +-- Daily aggregates (materialized or computed) +CREATE TABLE telemetry_daily ( + app_id UUID NOT NULL, + date DATE NOT NULL, + event_type VARCHAR(50) NOT NULL, + count BIGINT NOT NULL, + unique_devices BIGINT NOT NULL, + PRIMARY KEY (app_id, date, event_type) +); +``` + +### Indexes + +```sql +-- Developers +CREATE INDEX idx_developers_email ON developers(email); +CREATE INDEX idx_developers_oauth ON developers(oauth_provider, oauth_id); + +-- Apps +CREATE INDEX idx_apps_developer ON apps(developer_id); +CREATE INDEX idx_apps_package ON apps(package_id); +CREATE INDEX idx_apps_status ON apps(status); +CREATE INDEX idx_apps_search ON apps USING gin(to_tsvector('english', name || ' ' || COALESCE(description, ''))); + +-- Versions +CREATE INDEX idx_versions_app ON app_versions(app_id); +CREATE INDEX idx_versions_status ON app_versions(status); + +-- Telemetry +CREATE INDEX idx_telemetry_app ON telemetry_events(app_id, timestamp); +CREATE INDEX idx_telemetry_type ON telemetry_events(event_type, timestamp); + +-- Crashes +CREATE INDEX idx_crashes_app ON crash_reports(app_id, timestamp); +CREATE INDEX idx_crashes_type ON crash_reports(crash_type); +``` + +--- + +## Migration Strategy + +### Approach: Incremental Migrations + +``` +migrations/ +├── 001_create_developers.sql +├── 002_create_apps.sql +├── 003_create_versions.sql +├── 004_create_telemetry.sql +└── ... +``` + +### Tools +- **Go**: golang-migrate, goose +- **Node.js**: Prisma Migrate, Drizzle Kit +- **Rust**: sqlx migrate, refinery + +### Rollback Strategy +- Every migration has up/down +- Test rollbacks in staging +- Keep migrations small and focused + +--- + +## Backup Strategy + +### PostgreSQL +```bash +# Daily full backup +pg_dump -Fc $DATABASE_URL > backup_$(date +%Y%m%d).dump + +# Continuous WAL archiving to S3 +archive_command = 'aws s3 cp %p s3://backups/wal/%f' +``` + +### SQLite + Litestream +```yaml +# litestream.yml +dbs: + - path: /data/mosis.db + replicas: + - url: s3://backups/mosis + retention: 720h # 30 days +``` + +### Recovery Time Objectives +| Scenario | RTO | RPO | +|----------|-----|-----| +| Hardware failure | 1 hour | 5 minutes | +| Data corruption | 4 hours | 1 hour | +| Disaster recovery | 24 hours | 24 hours | + +--- + +## Recommendation + +### For MVP/Early Stage +**SQLite + Litestream** +- Simplest to operate +- Lowest cost +- Good enough for initial scale +- Easy migration to PostgreSQL later + +### For Production Scale +**PostgreSQL + TimescaleDB** +- Handles all data types well +- Time-series for telemetry +- Proven at scale +- Good tooling ecosystem + +### Hybrid (If needed later) +``` +PostgreSQL → Core data (developers, apps) +TimescaleDB → Telemetry (same cluster, extension) +Redis → Caching, rate limiting +``` + +--- + +## Deliverables + +- [ ] Final database selection +- [ ] Complete schema design +- [ ] Migration scripts +- [ ] Backup/restore procedures +- [ ] Connection pooling setup (if PostgreSQL) +- [ ] Monitoring queries + +--- + +## Open Questions + +1. Expected telemetry volume per day? +2. How long to retain raw telemetry? +3. Need for real-time analytics vs batch? +4. Multi-region requirements? + +--- + +## References + +- [PostgreSQL JSONB performance](https://www.postgresql.org/docs/current/datatype-json.html) +- [TimescaleDB vs InfluxDB](https://www.timescale.com/blog/timescaledb-vs-influxdb/) +- [Litestream documentation](https://litestream.io/) +- [SQLite at scale](https://www.sqlite.org/whentouse.html) diff --git a/DEV_PORTAL_M04_AUTH.md b/DEV_PORTAL_M04_AUTH.md new file mode 100644 index 0000000..c99973c --- /dev/null +++ b/DEV_PORTAL_M04_AUTH.md @@ -0,0 +1,429 @@ +# Milestone 4: Authentication System + +**Status**: Planning +**Goal**: Secure developer authentication and app signing infrastructure. + +--- + +## Overview + +Authentication covers two areas: +1. **Developer authentication** - Login to portal, API access +2. **App signing** - Package integrity and developer verification + +--- + +## Developer Authentication + +### Methods Required + +| Method | Use Case | Priority | +|--------|----------|----------| +| OAuth2 (GitHub) | Primary login | P0 | +| OAuth2 (Google) | Alternative login | P1 | +| Email + Password | Fallback | P2 | +| API Keys | CLI tools, CI/CD | P0 | + +### OAuth2 Flow + +``` +┌─────────┐ ┌─────────┐ ┌──────────┐ ┌─────────┐ +│ Browser │────►│ Portal │────►│ Provider │────►│ Callback│ +└─────────┘ └─────────┘ │(GitHub) │ └────┬────┘ + └──────────┘ │ + ▼ + ┌─────────────┐ + │ Create/Link │ + │ Account │ + └─────────────┘ +``` + +### OAuth2 Implementation + +#### GitHub OAuth + +``` +Authorization URL: https://github.com/login/oauth/authorize +Token URL: https://github.com/login/oauth/access_token +User Info: https://api.github.com/user +Scopes: read:user, user:email +``` + +#### Google OAuth + +``` +Authorization URL: https://accounts.google.com/o/oauth2/v2/auth +Token URL: https://oauth2.googleapis.com/token +User Info: https://www.googleapis.com/oauth2/v2/userinfo +Scopes: openid, email, profile +``` + +### Session Management + +#### JWT Tokens + +```json +{ + "sub": "dev_uuid", + "email": "dev@example.com", + "iat": 1704067200, + "exp": 1704153600, + "type": "access" +} +``` + +| Token Type | Lifetime | Storage | +|------------|----------|---------| +| Access Token | 1 hour | Memory/Cookie | +| Refresh Token | 30 days | HttpOnly Cookie | + +#### Token Refresh Flow + +``` +1. Access token expires +2. Client sends refresh token +3. Server validates refresh token +4. Issue new access + refresh tokens +5. Invalidate old refresh token +``` + +### API Key Authentication + +#### Key Format + +``` +mk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +│ │ └── 32 random bytes (base62) +│ └── Environment (live/test) +└── Prefix (mosis key) +``` + +#### Key Storage + +```sql +-- Only store hash, never the key itself +INSERT INTO api_keys ( + developer_id, + name, + key_hash, -- bcrypt or argon2 + key_prefix, -- "mk_live_abc" for display + permissions +) VALUES (...); +``` + +#### Key Permissions + +```json +{ + "permissions": [ + "apps:read", + "apps:write", + "versions:upload", + "telemetry:read" + ] +} +``` + +### Rate Limiting + +| Endpoint Category | Limit | Window | +|-------------------|-------|--------| +| Auth endpoints | 10 | 1 minute | +| API (authenticated) | 1000 | 1 hour | +| API (per key) | 100 | 1 minute | +| Upload | 10 | 1 hour | + +--- + +## App Signing + +### Key Generation + +#### Algorithm: Ed25519 + +``` +Private key: 32 bytes (256 bits) +Public key: 32 bytes (256 bits) +Signature: 64 bytes (512 bits) +``` + +#### Why Ed25519? +- Fast signing and verification +- Small key and signature sizes +- No configuration choices (secure by default) +- Widely supported + +### Key Generation Flow + +```bash +# Developer generates keypair locally +mosis keys generate + +# Output: +# Private key saved to: ~/.mosis/signing_key.pem +# Public key saved to: ~/.mosis/signing_key.pub +# +# Fingerprint: SHA256:xxxxx +# +# IMPORTANT: Keep your private key secure! +# Upload your public key to the developer portal. +``` + +### Key Format + +#### Private Key (PEM) + +``` +-----BEGIN PRIVATE KEY----- +MC4CAQAwBQYDK2VwBCIEIGxxxxx... +-----END PRIVATE KEY----- +``` + +#### Public Key (PEM) + +``` +-----BEGIN PUBLIC KEY----- +MCowBQYDK2VwAyEAxxxxx... +-----END PUBLIC KEY----- +``` + +### Signing Flow + +``` +1. Build package (ZIP all files) +2. Generate MANIFEST.MF with SHA-256 hashes +3. Sign MANIFEST.MF with private key +4. Store signature in META-INF/CERT.SIG +5. Include public key in META-INF/CERT.PEM +``` + +#### MANIFEST.MF Example + +``` +Manifest-Version: 1.0 +Created-By: mosis-cli 1.0.0 +Package-Id: com.developer.myapp +Version-Code: 1 + +Name: manifest.json +SHA-256-Digest: K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols= + +Name: assets/main.rml +SHA-256-Digest: uU0nuZNNPgilLlLX2n2r+sSE7+N6U4DukIj3rOLvzek= +``` + +### Verification Flow + +``` +1. Extract MANIFEST.MF from package +2. Extract CERT.SIG (signature) +3. Extract CERT.PEM (public key) +4. Verify signature of MANIFEST.MF using public key +5. Verify CERT.PEM matches registered developer key +6. Verify each file hash matches MANIFEST.MF entry +``` + +### Key Registration + +``` +Developer Portal: +├── Go to Settings > Signing Keys +├── Click "Add Key" +├── Paste public key (PEM format) +├── Verify fingerprint matches local +└── Key is now registered + +Server stores: +├── public_key (PEM text) +├── fingerprint (SHA256 of public key) +├── created_at +└── is_active +``` + +### Key Rotation + +``` +1. Generate new keypair +2. Register new public key in portal +3. Sign new versions with new key +4. (Optional) Revoke old key after transition period +``` + +### Trust Model + +``` +┌─────────────────────────────────────────────────┐ +│ Trust Chain │ +├─────────────────────────────────────────────────┤ +│ │ +│ Developer │ +│ │ │ +│ ▼ │ +│ Private Key ──signs──► Package │ +│ │ │ +│ ▼ │ +│ Public Key ──registered──► Portal │ +│ │ │ +│ ▼ │ +│ Portal ──verifies──► Signature │ +│ │ │ +│ ▼ │ +│ Device ──trusts──► Portal-verified packages │ +│ │ +└─────────────────────────────────────────────────┘ +``` + +--- + +## Security Considerations + +### Password Storage (if used) + +``` +Algorithm: Argon2id +Memory: 64 MB +Iterations: 3 +Parallelism: 4 +Salt: 16 bytes random +``` + +### Token Security + +- Access tokens: Short-lived, in-memory only +- Refresh tokens: HttpOnly, Secure, SameSite=Strict +- API keys: Hashed with bcrypt, shown once on creation + +### Key Security + +- Private keys never leave developer's machine +- Public keys verified via fingerprint +- Key compromise: Revoke immediately, re-sign apps + +### Audit Logging + +```sql +INSERT INTO auth_audit_log ( + developer_id, + action, -- login, logout, key_create, key_revoke + ip_address, + user_agent, + success, + failure_reason, + timestamp +) VALUES (...); +``` + +--- + +## API Endpoints + +### Authentication + +``` +POST /auth/oauth/github # Start GitHub OAuth +GET /auth/oauth/github/callback # GitHub callback +POST /auth/oauth/google # Start Google OAuth +GET /auth/oauth/google/callback # Google callback +POST /auth/refresh # Refresh tokens +POST /auth/logout # Invalidate tokens +GET /auth/me # Get current user +``` + +### API Keys + +``` +GET /api-keys # List keys +POST /api-keys # Create key +DELETE /api-keys/:id # Revoke key +``` + +### Signing Keys + +``` +GET /signing-keys # List keys +POST /signing-keys # Register key +DELETE /signing-keys/:id # Revoke key +GET /signing-keys/:id/verify # Verify a signature +``` + +--- + +## Implementation Libraries + +### Node.js + +```json +{ + "passport": "OAuth strategies", + "jose": "JWT handling", + "@noble/ed25519": "Ed25519 signing", + "argon2": "Password hashing" +} +``` + +### Go + +```go +import ( + "golang.org/x/oauth2" + "github.com/golang-jwt/jwt/v5" + "crypto/ed25519" + "golang.org/x/crypto/argon2" +) +``` + +### Rust + +```toml +[dependencies] +oauth2 = "4.4" +jsonwebtoken = "9" +ed25519-dalek = "2" +argon2 = "0.5" +``` + +--- + +## Deliverables + +- [ ] OAuth2 integration (GitHub) +- [ ] OAuth2 integration (Google) +- [ ] JWT token management +- [ ] API key generation and validation +- [ ] Ed25519 key generation tool +- [ ] Signature creation and verification +- [ ] Key registration API +- [ ] Audit logging + +--- + +## Test Cases + +| Test | Description | +|------|-------------| +| OAuthLogin | Complete OAuth flow | +| TokenRefresh | Refresh expired access token | +| InvalidToken | Reject tampered JWT | +| APIKeyAuth | Authenticate with API key | +| KeyGeneration | Generate valid Ed25519 keypair | +| SignPackage | Sign and verify package | +| InvalidSignature | Reject tampered package | +| KeyRevocation | Revoked key fails verification | + +--- + +## Open Questions + +1. Support for hardware security keys (YubiKey)? +2. Multi-factor authentication for portal? +3. Team accounts with role-based access? +4. Key escrow for enterprise customers? + +--- + +## References + +- [OAuth 2.0 RFC 6749](https://tools.ietf.org/html/rfc6749) +- [JWT RFC 7519](https://tools.ietf.org/html/rfc7519) +- [Ed25519 paper](https://ed25519.cr.yp.to/ed25519-20110926.pdf) +- [OWASP Authentication Cheatsheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html) diff --git a/DEV_PORTAL_M05_FRONTEND.md b/DEV_PORTAL_M05_FRONTEND.md new file mode 100644 index 0000000..746f02e --- /dev/null +++ b/DEV_PORTAL_M05_FRONTEND.md @@ -0,0 +1,464 @@ +# Milestone 5: Developer Portal Frontend + +**Status**: Planning +**Goal**: Web interface for developer account and app management. + +--- + +## Overview + +The developer portal is the primary interface for developers to manage their accounts, create apps, submit versions, and view analytics. + +--- + +## Pages Required + +### Public Pages + +| Page | URL | Purpose | +|------|-----|---------| +| Landing | `/` | Marketing, sign up CTA | +| Sign In | `/login` | OAuth + password login | +| Sign Up | `/register` | Create account | +| Docs | `/docs/*` | Documentation (separate site?) | + +### Authenticated Pages + +| Page | URL | Purpose | +|------|-----|---------| +| Dashboard | `/dashboard` | App list, quick stats | +| App Details | `/apps/:id` | Single app overview | +| App Settings | `/apps/:id/settings` | Edit app metadata | +| App Versions | `/apps/:id/versions` | Version history | +| Submit Version | `/apps/:id/versions/new` | Upload new version | +| App Analytics | `/apps/:id/analytics` | Telemetry dashboard | +| Create App | `/apps/new` | New app wizard | +| API Keys | `/settings/keys` | Manage API keys | +| Signing Keys | `/settings/signing` | Manage signing keys | +| Profile | `/settings/profile` | Account settings | + +--- + +## Wireframes + +### Dashboard + +``` +┌─────────────────────────────────────────────────────────────┐ +│ [Logo] Dashboard Apps Docs Settings [Avatar ▼] │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Welcome back, Developer! │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Total Apps │ │ Downloads │ │ Active Users│ │ +│ │ 12 │ │ 45,230 │ │ 1,234 │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ Your Apps [+ New App] │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ [Icon] My Calculator v1.2.0 Published 1.2K ↓ │ │ +│ │ [Icon] Notes App v2.0.1 Published 5.4K ↓ │ │ +│ │ [Icon] Weather Widget v0.9.0 In Review --- │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### App Details + +``` +┌─────────────────────────────────────────────────────────────┐ +│ ← Back to Apps │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ [Icon] My Calculator │ +│ com.developer.calculator │ +│ ● Published │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Overview │ │ Versions │ │Analytics │ │ Settings │ │ +│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +│ Latest Version: 1.2.0 [Submit New Version] │ +│ Published: Jan 15, 2024 │ +│ Downloads: 1,234 │ +│ │ +│ ┌────────────────────────────────────────┐ │ +│ │ Downloads over time │ │ +│ │ [Chart: line graph] │ │ +│ └────────────────────────────────────────┘ │ +│ │ +│ Recent Crashes [View All →] │ +│ • attempt to index nil (v1.2.0) - 23 reports │ +│ • memory limit exceeded (v1.1.0) - 5 reports │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Submit Version + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Submit New Version - My Calculator │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Step 2 of 3: Upload Package │ +│ ○ Details ● Upload ○ Review │ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ │ │ +│ │ ┌─────────┐ │ │ +│ │ │ .mosis │ Drop your package here │ │ +│ │ │ 📦 │ or click to browse │ │ +│ │ └─────────┘ │ │ +│ │ │ │ +│ │ Max size: 50 MB │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ ✓ Package validated │ +│ ✓ Signature verified │ +│ ✓ Permissions: storage, network │ +│ │ +│ [Back] [Continue to Review] │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Tech Stack Options + +### Option A: Next.js 14 + +``` +Framework: Next.js 14 (App Router) +UI: shadcn/ui + Tailwind CSS +State: React Query + Zustand +Forms: React Hook Form + Zod +Charts: Recharts or Tremor +Auth: NextAuth.js +``` + +| Pros | Cons | +|------|------| +| SSR for SEO landing page | Complexity | +| Great developer experience | React knowledge required | +| Full-stack capability | Heavier bundle | +| Large ecosystem | | + +### Option B: SvelteKit + +``` +Framework: SvelteKit +UI: Skeleton UI or custom +State: Svelte stores +Forms: Superforms +Charts: Chart.js or LayerCake +Auth: Lucia +``` + +| Pros | Cons | +|------|------| +| Fast, small bundles | Smaller ecosystem | +| Simple state management | Fewer UI libraries | +| Great DX | Less hiring pool | + +### Option C: Astro + React Islands + +``` +Framework: Astro +UI: React components (islands) +State: Nanostores +Forms: React Hook Form +Charts: Recharts +Auth: Custom +``` + +| Pros | Cons | +|------|------| +| Ultra-fast static pages | More setup | +| Partial hydration | Newer approach | +| Use React where needed | Less documented patterns | + +### Option D: htmx + Go Templates + +``` +Framework: Go templates + htmx +UI: Tailwind CSS +State: Server-side +Forms: Native HTML +Charts: Chart.js +Auth: Server sessions +``` + +| Pros | Cons | +|------|------| +| Simple, fast | Limited interactivity | +| No JS framework | Less polished UX | +| Server-rendered | Complex UI harder | + +--- + +## UI Component Library + +### Option A: shadcn/ui + +``` +Base: Radix UI primitives +Styling: Tailwind CSS +Approach: Copy-paste components +``` + +| Pros | Cons | +|------|------| +| High quality | React only | +| Full control | Manual updates | +| Accessible | | + +### Option B: Tailwind UI + +``` +Base: Headless UI +Styling: Tailwind CSS +Approach: Copy-paste templates +``` + +| Pros | Cons | +|------|------| +| Beautiful designs | Paid ($299) | +| Production-ready | Templates, not components | + +### Option C: Mantine + +``` +Base: Custom components +Styling: CSS-in-JS or CSS +Approach: npm package +``` + +| Pros | Cons | +|------|------| +| Complete solution | Opinionated | +| Many components | Larger bundle | +| Good docs | | + +--- + +## Key Features + +### File Upload + +```typescript +// Drag and drop with progress + + +

Drop your package here

+ +``` + +### Real-time Validation + +```typescript +// Validate package client-side before upload +async function validatePackage(file: File) { + const zip = await JSZip.loadAsync(file); + + // Check manifest + const manifest = await zip.file('manifest.json')?.async('text'); + if (!manifest) throw new Error('Missing manifest.json'); + + const parsed = JSON.parse(manifest); + ManifestSchema.parse(parsed); + + // Check required files + const entry = parsed.entry; + if (!zip.file(entry)) { + throw new Error(`Entry file not found: ${entry}`); + } + + return parsed; +} +``` + +### Analytics Dashboard + +```typescript +// Recharts example + + + + + + +``` + +--- + +## State Management + +### Server State (React Query) + +```typescript +// Fetch apps +const { data: apps, isLoading } = useQuery({ + queryKey: ['apps'], + queryFn: () => api.get('/apps'), +}); + +// Create app mutation +const createApp = useMutation({ + mutationFn: (data) => api.post('/apps', data), + onSuccess: () => { + queryClient.invalidateQueries(['apps']); + }, +}); +``` + +### Client State (Zustand) + +```typescript +// UI state +const useStore = create((set) => ({ + sidebarOpen: true, + toggleSidebar: () => set((s) => ({ sidebarOpen: !s.sidebarOpen })), +})); +``` + +--- + +## Authentication Flow + +### Login Page + +```tsx +export default function LoginPage() { + return ( +
+ + +

Sign in to Mosis

+ + + + + +
+ + + +
+ + +
+ ); +} +``` + +### Protected Routes + +```tsx +// Middleware (Next.js) +export function middleware(request: NextRequest) { + const token = request.cookies.get('token'); + + if (!token && request.nextUrl.pathname.startsWith('/dashboard')) { + return NextResponse.redirect(new URL('/login', request.url)); + } +} +``` + +--- + +## Responsive Design + +### Breakpoints + +| Size | Width | Target | +|------|-------|--------| +| sm | 640px | Mobile landscape | +| md | 768px | Tablet | +| lg | 1024px | Desktop | +| xl | 1280px | Large desktop | + +### Mobile Navigation + +```tsx +// Hamburger menu for mobile + + + + + + + + +``` + +--- + +## Accessibility + +### Requirements + +- [ ] Keyboard navigation +- [ ] Screen reader support +- [ ] Color contrast (WCAG AA) +- [ ] Focus indicators +- [ ] Alt text for images +- [ ] Form labels +- [ ] Error announcements + +### Testing + +```bash +# Lighthouse audit +npx lighthouse http://localhost:3000 --view + +# axe-core +npm install @axe-core/react +``` + +--- + +## Deliverables + +- [ ] Framework selection +- [ ] UI component library selection +- [ ] Design system (colors, typography) +- [ ] Page wireframes +- [ ] Authentication flow +- [ ] Dashboard implementation +- [ ] App management pages +- [ ] Version submission flow +- [ ] Settings pages +- [ ] Responsive design +- [ ] Accessibility audit + +--- + +## Open Questions + +1. Dark mode support? +2. Internationalization (i18n)? +3. Custom domain for docs vs integrated? +4. Email notifications UI? + +--- + +## References + +- [shadcn/ui](https://ui.shadcn.com/) +- [Next.js App Router](https://nextjs.org/docs/app) +- [Tailwind CSS](https://tailwindcss.com/) +- [React Query](https://tanstack.com/query/latest) diff --git a/DEV_PORTAL_M06_API.md b/DEV_PORTAL_M06_API.md new file mode 100644 index 0000000..2cf9b29 --- /dev/null +++ b/DEV_PORTAL_M06_API.md @@ -0,0 +1,604 @@ +# Milestone 6: App Store Backend API + +**Status**: Planning +**Goal**: REST API for app submission, review, and distribution. + +--- + +## Overview + +The backend API serves the developer portal, CLI tools, and device-side app management. It handles authentication, app lifecycle, file storage, and telemetry ingestion. + +--- + +## API Design Principles + +1. **RESTful** - Standard HTTP methods and status codes +2. **JSON** - Request and response bodies in JSON +3. **Versioned** - `/v1/` prefix for breaking changes +4. **Consistent** - Same patterns across all endpoints +5. **Documented** - OpenAPI specification + +--- + +## Base URL + +``` +Production: https://api.mosis.dev/v1 +Staging: https://api.staging.mosis.dev/v1 +Local: http://localhost:8080/v1 +``` + +--- + +## Authentication + +### Headers + +``` +Authorization: Bearer +# or +X-API-Key: mk_live_xxxxxxxx +``` + +### Scopes + +| Scope | Description | +|-------|-------------| +| `apps:read` | Read app metadata | +| `apps:write` | Create/update apps | +| `versions:upload` | Upload new versions | +| `versions:publish` | Publish versions | +| `telemetry:read` | Read analytics | +| `keys:manage` | Manage API keys | + +--- + +## Endpoints + +### Authentication + +```yaml +POST /v1/auth/oauth/github: + summary: Start GitHub OAuth flow + response: { redirect_url: string } + +GET /v1/auth/oauth/github/callback: + summary: GitHub OAuth callback + query: + code: string + state: string + response: { access_token, refresh_token, user } + +POST /v1/auth/oauth/google: + summary: Start Google OAuth flow + response: { redirect_url: string } + +GET /v1/auth/oauth/google/callback: + summary: Google OAuth callback + +POST /v1/auth/refresh: + summary: Refresh access token + body: { refresh_token: string } + response: { access_token, refresh_token } + +POST /v1/auth/logout: + summary: Invalidate tokens + auth: required + +GET /v1/auth/me: + summary: Get current user + auth: required + response: Developer +``` + +### Apps + +```yaml +GET /v1/apps: + summary: List developer's apps + auth: required + query: + status: draft | published | suspended + page: number + limit: number + response: { apps: App[], total: number } + +POST /v1/apps: + summary: Create new app + auth: required + body: + package_id: string # com.developer.appname + name: string + description?: string + category?: string + response: App + +GET /v1/apps/:id: + summary: Get app details + auth: required + response: App + +PATCH /v1/apps/:id: + summary: Update app metadata + auth: required + body: + name?: string + description?: string + category?: string + tags?: string[] + response: App + +DELETE /v1/apps/:id: + summary: Delete app (if no published versions) + auth: required + response: { success: true } +``` + +### App Versions + +```yaml +GET /v1/apps/:id/versions: + summary: List app versions + auth: required + query: + status: draft | review | approved | published | rejected + page: number + limit: number + response: { versions: AppVersion[], total: number } + +POST /v1/apps/:id/versions: + summary: Create new version (get upload URL) + auth: required + body: + version_name: string # 1.0.0 + version_code: number # 1 + release_notes?: string + response: + version: AppVersion + upload_url: string # Presigned S3 URL + upload_expires: string # ISO timestamp + +PUT /v1/apps/:id/versions/:vid/upload-complete: + summary: Mark upload as complete, trigger validation + auth: required + response: AppVersion + +GET /v1/apps/:id/versions/:vid: + summary: Get version details + auth: required + response: AppVersion + +POST /v1/apps/:id/versions/:vid/submit: + summary: Submit version for review + auth: required + response: AppVersion # status: review + +POST /v1/apps/:id/versions/:vid/publish: + summary: Publish approved version + auth: required + response: AppVersion # status: published + +DELETE /v1/apps/:id/versions/:vid: + summary: Delete draft version + auth: required + response: { success: true } +``` + +### Public App Store + +```yaml +GET /v1/store/apps: + summary: Browse/search published apps + auth: none + query: + q: string # Search query + category: string + sort: popular | recent | name + page: number + limit: number + response: { apps: PublicApp[], total: number } + +GET /v1/store/apps/:package_id: + summary: Get app store listing + auth: none + response: PublicApp + +GET /v1/store/apps/:package_id/download: + summary: Get download URL for latest version + auth: none (or device token) + response: + download_url: string + version: string + size: number + signature: string + +GET /v1/store/apps/:package_id/versions/:version_code/download: + summary: Get download URL for specific version + auth: none + response: { download_url, version, size, signature } +``` + +### API Keys + +```yaml +GET /v1/keys: + summary: List API keys + auth: required + response: { keys: APIKey[] } + +POST /v1/keys: + summary: Create API key + auth: required + body: + name: string + permissions: string[] + expires_at?: string + response: + key: APIKey + secret: string # Only shown once! + +DELETE /v1/keys/:id: + summary: Revoke API key + auth: required + response: { success: true } +``` + +### Signing Keys + +```yaml +GET /v1/signing-keys: + summary: List signing keys + auth: required + response: { keys: SigningKey[] } + +POST /v1/signing-keys: + summary: Register signing key + auth: required + body: + name: string + public_key: string # PEM format + response: SigningKey + +DELETE /v1/signing-keys/:id: + summary: Revoke signing key + auth: required + response: { success: true } +``` + +### Telemetry + +```yaml +POST /v1/telemetry/events: + summary: Submit telemetry events (batch) + auth: device token or API key + body: + events: + - app_id: string + event_type: string + event_data: object + timestamp: string + response: { received: number } + +POST /v1/telemetry/crash: + summary: Submit crash report + auth: device token or API key + body: + app_id: string + app_version: string + crash_type: string + message: string + stack_trace: string + context: object + timestamp: string + response: { id: string } + +GET /v1/apps/:id/analytics: + summary: Get app analytics + auth: required + query: + start_date: string + end_date: string + metrics: downloads | active_users | crashes + response: + data: + - date: string + downloads: number + active_users: number + crashes: number + +GET /v1/apps/:id/crashes: + summary: Get crash reports + auth: required + query: + version?: string + page: number + limit: number + response: { crashes: CrashReport[], total: number } +``` + +--- + +## Data Models + +### Developer + +```typescript +interface Developer { + id: string; + email: string; + name: string; + avatar_url?: string; + verified: boolean; + created_at: string; + updated_at: string; +} +``` + +### App + +```typescript +interface App { + id: string; + package_id: string; + name: string; + description?: string; + category?: string; + tags: string[]; + status: 'draft' | 'published' | 'suspended'; + icon_url?: string; + latest_version?: AppVersion; + created_at: string; + updated_at: string; +} +``` + +### AppVersion + +```typescript +interface AppVersion { + id: string; + app_id: string; + version_name: string; + version_code: number; + package_url?: string; + package_size?: number; + signature?: string; + permissions: string[]; + min_mosis_version?: string; + release_notes?: string; + status: 'draft' | 'uploading' | 'validating' | 'review' | 'approved' | 'published' | 'rejected'; + review_notes?: string; + published_at?: string; + created_at: string; +} +``` + +### PublicApp + +```typescript +interface PublicApp { + package_id: string; + name: string; + description?: string; + category?: string; + tags: string[]; + icon_url?: string; + author_name: string; + latest_version: string; + download_count: number; + rating?: number; + created_at: string; + updated_at: string; +} +``` + +### APIKey + +```typescript +interface APIKey { + id: string; + name: string; + key_prefix: string; // "mk_live_abc..." + permissions: string[]; + last_used_at?: string; + expires_at?: string; + created_at: string; +} +``` + +### CrashReport + +```typescript +interface CrashReport { + id: string; + app_id: string; + app_version: string; + crash_type: string; + message: string; + stack_trace: string; + context: object; + occurrences: number; + first_seen: string; + last_seen: string; +} +``` + +--- + +## Error Handling + +### Error Response Format + +```json +{ + "error": { + "code": "VALIDATION_ERROR", + "message": "Invalid package_id format", + "details": { + "field": "package_id", + "constraint": "Must match pattern: ^[a-z][a-z0-9]*(\\.[a-z][a-z0-9]*)+$" + } + } +} +``` + +### Error Codes + +| Code | HTTP Status | Description | +|------|-------------|-------------| +| `UNAUTHORIZED` | 401 | Missing or invalid auth | +| `FORBIDDEN` | 403 | Insufficient permissions | +| `NOT_FOUND` | 404 | Resource not found | +| `VALIDATION_ERROR` | 400 | Invalid request body | +| `CONFLICT` | 409 | Resource already exists | +| `RATE_LIMITED` | 429 | Too many requests | +| `INTERNAL_ERROR` | 500 | Server error | + +--- + +## Rate Limiting + +### Headers + +``` +X-RateLimit-Limit: 1000 +X-RateLimit-Remaining: 999 +X-RateLimit-Reset: 1704067200 +``` + +### Limits + +| Endpoint Category | Limit | Window | +|-------------------|-------|--------| +| Auth | 10 | 1 minute | +| Read | 1000 | 1 hour | +| Write | 100 | 1 hour | +| Upload | 10 | 1 hour | +| Telemetry | 10000 | 1 hour | + +--- + +## Pagination + +### Request + +``` +GET /v1/apps?page=2&limit=20 +``` + +### Response + +```json +{ + "apps": [...], + "pagination": { + "page": 2, + "limit": 20, + "total": 45, + "total_pages": 3 + } +} +``` + +--- + +## Webhooks (Future) + +```yaml +POST /v1/webhooks: + summary: Register webhook + body: + url: string + events: string[] # version.published, crash.new + secret: string + +Webhook Payload: + headers: + X-Mosis-Signature: sha256=xxx + body: + event: string + data: object + timestamp: string +``` + +--- + +## OpenAPI Specification + +Full OpenAPI 3.0 spec will be generated and hosted at: +- `https://api.mosis.dev/v1/openapi.json` +- `https://api.mosis.dev/v1/docs` (Swagger UI) + +--- + +## Implementation Structure + +``` +src/ +├── main.go (or index.ts) +├── api/ +│ ├── routes.go +│ ├── middleware/ +│ │ ├── auth.go +│ │ ├── ratelimit.go +│ │ └── logging.go +│ └── handlers/ +│ ├── auth.go +│ ├── apps.go +│ ├── versions.go +│ ├── store.go +│ ├── keys.go +│ └── telemetry.go +├── service/ +│ ├── app_service.go +│ ├── version_service.go +│ ├── auth_service.go +│ └── storage_service.go +├── repository/ +│ ├── app_repo.go +│ ├── version_repo.go +│ └── developer_repo.go +├── domain/ +│ ├── app.go +│ ├── version.go +│ └── developer.go +└── pkg/ + ├── validator/ + └── crypto/ +``` + +--- + +## Deliverables + +- [ ] OpenAPI specification +- [ ] Authentication middleware +- [ ] Rate limiting middleware +- [ ] Auth endpoints +- [ ] Apps CRUD endpoints +- [ ] Versions endpoints with upload flow +- [ ] Store public endpoints +- [ ] API keys management +- [ ] Signing keys management +- [ ] Telemetry ingestion +- [ ] Error handling +- [ ] Request validation +- [ ] Integration tests + +--- + +## Open Questions + +1. GraphQL alongside REST? +2. WebSocket for real-time review status? +3. Batch operations for bulk updates? +4. API versioning strategy (URL vs header)? + +--- + +## References + +- [REST API Design Guidelines](https://github.com/microsoft/api-guidelines) +- [OpenAPI 3.0 Specification](https://swagger.io/specification/) +- [HTTP Status Codes](https://httpstatuses.com/) diff --git a/DEV_PORTAL_M07_STORAGE.md b/DEV_PORTAL_M07_STORAGE.md new file mode 100644 index 0000000..ef85dad --- /dev/null +++ b/DEV_PORTAL_M07_STORAGE.md @@ -0,0 +1,451 @@ +# Milestone 7: CDN & Storage + +**Status**: Planning +**Goal**: Scalable storage for app packages and assets with global distribution. + +--- + +## Overview + +Storage handles app packages, icons, screenshots, and serves downloads to devices worldwide. Must be cost-effective, fast, and reliable. + +--- + +## Requirements + +### Functional + +- Store app packages (.mosis files) +- Store icons (multiple sizes) +- Store screenshots (optional) +- Serve downloads globally +- Support presigned upload URLs +- Version retention (keep old versions) + +### Non-Functional + +| Requirement | Target | +|-------------|--------| +| Upload speed | < 30s for 50MB | +| Download latency | < 100ms (p50) | +| Availability | 99.9% | +| Durability | 99.999999999% (11 nines) | +| Cost | Minimize egress fees | + +--- + +## Storage Structure + +``` +/packages/ + /{developer_id}/ + /{app_id}/ + /{version_code}/ + package.mosis + manifest.json (extracted for quick access) + +/assets/ + /{app_id}/ + icon-32.png + icon-64.png + icon-128.png + screenshots/ + 1.png + 2.png + 3.png + +/temp/ + /{upload_id}/ + package.mosis (pending validation) +``` + +--- + +## Options Analysis + +### Option A: Cloudflare R2 + +``` +Storage: Object storage (S3-compatible) +CDN: Cloudflare network (automatic) +Egress: FREE (zero egress fees) +Pricing: $0.015/GB storage, $4.50/million requests +``` + +| Pros | Cons | +|------|------| +| No egress fees | Newer service | +| Global CDN included | Fewer regions than S3 | +| S3-compatible API | Less tooling | +| Workers integration | | + +#### Cost Estimate (10K apps, 100GB) + +| Component | Monthly Cost | +|-----------|--------------| +| Storage (100GB) | $1.50 | +| Requests (1M) | $4.50 | +| Egress | $0 | +| **Total** | **~$6** | + +--- + +### Option B: AWS S3 + CloudFront + +``` +Storage: S3 Standard +CDN: CloudFront +Egress: $0.085-0.12/GB (varies by region) +Pricing: $0.023/GB storage +``` + +| Pros | Cons | +|------|------| +| Most mature | Egress costs add up | +| Best tooling | Complex pricing | +| All regions | Need CloudFront config | +| IAM integration | | + +#### Cost Estimate (10K apps, 100GB, 1TB egress) + +| Component | Monthly Cost | +|-----------|--------------| +| Storage (100GB) | $2.30 | +| CloudFront (1TB) | $85 | +| Requests | ~$5 | +| **Total** | **~$92** | + +--- + +### Option C: Backblaze B2 + Cloudflare + +``` +Storage: Backblaze B2 +CDN: Cloudflare (free egress via Bandwidth Alliance) +Egress: FREE (through Cloudflare) +Pricing: $0.005/GB storage +``` + +| Pros | Cons | +|------|------| +| Cheapest storage | Two services to manage | +| Free egress via CF | B2 API slightly different | +| Good reliability | Need CF proxy setup | + +#### Cost Estimate (10K apps, 100GB) + +| Component | Monthly Cost | +|-----------|--------------| +| Storage (100GB) | $0.50 | +| Egress (via CF) | $0 | +| Requests | ~$0.40 | +| **Total** | **~$1** | + +--- + +### Option D: Self-hosted MinIO + +``` +Storage: MinIO on VPS +CDN: Cloudflare proxy +Egress: VPS bandwidth +Pricing: VPS cost only +``` + +| Pros | Cons | +|------|------| +| Full control | Ops overhead | +| S3-compatible | Need to manage | +| Predictable cost | Scaling complexity | + +#### Cost Estimate + +| Component | Monthly Cost | +|-----------|--------------| +| VPS (500GB SSD) | $20-50 | +| Cloudflare | $0 (free tier) | +| **Total** | **~$20-50** | + +--- + +## Recommendation + +**Primary: Cloudflare R2** +- Zero egress fees (biggest cost saver) +- S3-compatible (easy migration) +- Built-in global CDN +- Good enough tooling + +**Fallback: Backblaze B2 + Cloudflare** +- If R2 has issues +- Even cheaper storage +- Slightly more setup + +--- + +## Upload Flow + +### Presigned URL Approach + +``` +┌────────┐ ┌─────────┐ ┌─────────┐ +│ Client │───►│ API │───►│ R2 │ +└────────┘ └─────────┘ └─────────┘ + │ │ │ + │ 1. Request upload URL │ + │◄─────────────┤ │ + │ (presigned URL) │ + │ │ + │ 2. Upload directly │ + │────────────────────────────►│ + │ │ + │ 3. Notify complete │ + │─────────────►│ │ + │ │ 4. Validate │ + │ │─────────────►│ + │ │ │ + │ 5. Confirm │ │ + │◄─────────────┤ │ +``` + +### API Implementation + +```go +// 1. Request upload URL +func CreateVersion(c *gin.Context) { + // Create version record + version := Version{ + AppID: appID, + VersionCode: req.VersionCode, + Status: "uploading", + } + db.Create(&version) + + // Generate presigned URL + key := fmt.Sprintf("temp/%s/package.mosis", version.ID) + url, err := r2.PresignPut(key, 15*time.Minute) + + c.JSON(200, gin.H{ + "version": version, + "upload_url": url, + "expires": time.Now().Add(15 * time.Minute), + }) +} + +// 3. Upload complete notification +func UploadComplete(c *gin.Context) { + // Move from temp to final location + tempKey := fmt.Sprintf("temp/%s/package.mosis", versionID) + finalKey := fmt.Sprintf("packages/%s/%s/%d/package.mosis", + developerID, appID, versionCode) + + r2.Copy(tempKey, finalKey) + r2.Delete(tempKey) + + // Update version status + version.Status = "validating" + version.PackageURL = finalKey + + // Trigger async validation + queue.Publish("validate-package", version.ID) +} +``` + +--- + +## Download Flow + +### Public Downloads + +```go +// Get download URL (short-lived) +func GetDownloadURL(c *gin.Context) { + version := getLatestPublishedVersion(packageID) + + // Generate presigned download URL (1 hour) + url, _ := r2.PresignGet(version.PackageURL, 1*time.Hour) + + c.JSON(200, gin.H{ + "download_url": url, + "version": version.VersionName, + "size": version.PackageSize, + "signature": version.Signature, + }) +} +``` + +### CDN Caching + +``` +Cache-Control: public, max-age=86400 +``` + +- Packages are immutable (version code = unique) +- Cache aggressively at edge +- Invalidate only if package is pulled + +--- + +## Icon/Screenshot Handling + +### Upload + +```go +// Icons uploaded with app creation/update +func UploadIcon(c *gin.Context) { + file, _ := c.FormFile("icon") + + // Validate dimensions + img, _ := png.Decode(file) + if img.Bounds().Dx() != img.Bounds().Dy() { + return error("Icon must be square") + } + + // Generate multiple sizes + sizes := []int{32, 64, 128} + for _, size := range sizes { + resized := resize(img, size, size) + key := fmt.Sprintf("assets/%s/icon-%d.png", appID, size) + r2.Put(key, resized) + } +} +``` + +### Serving + +``` +https://cdn.mosis.dev/assets/{app_id}/icon-64.png +``` + +- Public read access +- Long cache TTL (icons rarely change) +- Cloudflare image optimization (optional) + +--- + +## Retention Policy + +### Packages + +| Status | Retention | +|--------|-----------| +| Published (current) | Forever | +| Published (old) | 1 year | +| Draft | 30 days | +| Rejected | 7 days | +| Failed validation | 24 hours | + +### Temp Uploads + +- Delete after 1 hour if not completed +- Lifecycle rule on `/temp/` prefix + +### Implementation + +```yaml +# R2 Lifecycle Rules +rules: + - prefix: "temp/" + expiration_days: 1 + + - prefix: "packages/" + tags: + status: draft + expiration_days: 30 +``` + +--- + +## Backup Strategy + +### R2 Built-in + +- 11 nines durability +- Automatic replication within region +- No additional backup needed for packages + +### Metadata Backup + +- Package metadata in PostgreSQL +- PostgreSQL backups cover this +- Can regenerate URLs from DB + +--- + +## Monitoring + +### Metrics to Track + +| Metric | Source | +|--------|--------| +| Upload success rate | API logs | +| Download latency | Cloudflare analytics | +| Storage usage | R2 dashboard | +| Bandwidth | R2 dashboard | +| Error rate | API logs | + +### Alerts + +- Upload failure rate > 5% +- Download error rate > 1% +- Storage > 80% of budget + +--- + +## Security + +### Access Control + +``` +Packages: + - Write: API only (presigned URLs) + - Read: Public (presigned URLs) + +Assets: + - Write: API only + - Read: Public (direct CDN) + +Temp: + - Write: Presigned URLs (15 min) + - Read: None (API internal only) +``` + +### Signed URLs + +```go +// Presigned URL with expiration +url := r2.PresignPut(key, 15*time.Minute, PutOptions{ + ContentType: "application/octet-stream", + ContentLength: maxSize, +}) +``` + +--- + +## Deliverables + +- [ ] R2 bucket setup +- [ ] Presigned URL generation +- [ ] Upload flow implementation +- [ ] Download URL generation +- [ ] Icon/screenshot upload +- [ ] Lifecycle rules for cleanup +- [ ] Monitoring dashboard +- [ ] Cost tracking + +--- + +## Open Questions + +1. Multi-region storage for lower latency? +2. Package compression (gzip/brotli)? +3. Delta updates storage structure? +4. Screenshot requirements (dimensions, count)? + +--- + +## References + +- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/) +- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html) +- [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html) diff --git a/DEV_PORTAL_M08_TELEMETRY.md b/DEV_PORTAL_M08_TELEMETRY.md new file mode 100644 index 0000000..045901f --- /dev/null +++ b/DEV_PORTAL_M08_TELEMETRY.md @@ -0,0 +1,503 @@ +# Milestone 8: Telemetry System + +**Status**: Planning +**Goal**: Collect app usage analytics and crash reports while respecting privacy. + +--- + +## Overview + +Telemetry provides developers with insights into app usage, performance, and crashes. Must balance usefulness with user privacy. + +--- + +## Privacy Principles + +1. **Minimal collection** - Only what's necessary +2. **No PII by default** - Anonymized device IDs +3. **Transparency** - Users know what's collected +4. **Opt-out available** - Users can disable +5. **Data retention limits** - Auto-delete old data +6. **GDPR compliance** - Export/delete on request + +--- + +## Event Types + +### Automatic Events (Default) + +| Event | Description | Data | +|-------|-------------|------| +| `app_start` | App launched | version, mosis_version | +| `app_stop` | App closed | duration_seconds | +| `app_crash` | Unhandled error | crash_type, message | +| `lua_error` | Lua runtime error | message, stack (no user data) | + +### Performance Events (Default) + +| Event | Description | Data | +|-------|-------------|------| +| `perf_frame` | Frame time (sampled) | avg_ms, p95_ms | +| `perf_memory` | Memory usage | used_mb, limit_mb | +| `perf_startup` | Startup time | duration_ms | + +### Usage Events (Opt-in) + +| Event | Description | Data | +|-------|-------------|------| +| `screen_view` | Screen navigation | screen_name | +| `button_click` | UI interaction | element_id | +| `feature_used` | Feature usage | feature_name | + +--- + +## Data Schema + +### Event Payload + +```json +{ + "app_id": "com.developer.myapp", + "app_version": "1.2.0", + "mosis_version": "1.0.0", + "device_id": "sha256_hashed_id", + "session_id": "uuid", + "events": [ + { + "type": "app_start", + "timestamp": "2024-01-15T10:30:00Z", + "data": {} + }, + { + "type": "screen_view", + "timestamp": "2024-01-15T10:30:05Z", + "data": { + "screen_name": "home" + } + } + ] +} +``` + +### Crash Report Payload + +```json +{ + "app_id": "com.developer.myapp", + "app_version": "1.2.0", + "mosis_version": "1.0.0", + "device_id": "sha256_hashed_id", + "timestamp": "2024-01-15T10:35:00Z", + "crash": { + "type": "lua_error", + "message": "attempt to index nil value 'user'", + "stack_trace": "main.lua:42: in function 'loadUser'\nmain.lua:15: in main chunk", + "context": { + "screen": "profile.rml", + "memory_mb": 45, + "uptime_seconds": 300 + } + } +} +``` + +### Device ID Hashing + +```lua +-- On device +local raw_id = get_android_id() -- or similar +local hashed = sha256(raw_id .. "mosis_salt_" .. app_id) +-- Result: "a3f2b1c4d5e6..." + +-- Cannot reverse to original device ID +-- Different per app (can't track across apps) +``` + +--- + +## Collection Architecture + +``` +┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ +│ Device │────►│ Batch │────►│ API │────►│ Storage │ +│ │ │ Queue │ │ │ │ │ +└──────────┘ └──────────┘ └──────────┘ └──────────┘ + │ + │ Every 60s or + │ on app close + ▼ + ┌──────────┐ + │ Upload │ + └──────────┘ +``` + +### Client-Side Batching + +```lua +-- TelemetryManager on device +local events = {} +local last_flush = os.time() + +function track(event_type, data) + if not telemetry_enabled then return end + + table.insert(events, { + type = event_type, + timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ"), + data = data or {} + }) + + -- Flush if batch is large or time elapsed + if #events >= 50 or (os.time() - last_flush) > 60 then + flush() + end +end + +function flush() + if #events == 0 then return end + + local payload = { + app_id = APP_ID, + app_version = APP_VERSION, + device_id = HASHED_DEVICE_ID, + events = events + } + + -- Async HTTP POST + http.post(TELEMETRY_URL, json.encode(payload)) + + events = {} + last_flush = os.time() +end +``` + +--- + +## Storage Options + +### Option A: PostgreSQL + TimescaleDB + +```sql +-- Hypertable for time-series data +CREATE TABLE telemetry_events ( + time TIMESTAMPTZ NOT NULL, + app_id TEXT NOT NULL, + device_id TEXT NOT NULL, + session_id TEXT, + event_type TEXT NOT NULL, + event_data JSONB, + app_version TEXT, + mosis_version TEXT +); + +SELECT create_hypertable('telemetry_events', 'time'); + +-- Continuous aggregate for daily stats +CREATE MATERIALIZED VIEW daily_stats +WITH (timescaledb.continuous) AS +SELECT + time_bucket('1 day', time) AS day, + app_id, + event_type, + COUNT(*) as count, + COUNT(DISTINCT device_id) as unique_devices +FROM telemetry_events +GROUP BY day, app_id, event_type; +``` + +### Option B: ClickHouse + +```sql +CREATE TABLE telemetry_events ( + timestamp DateTime, + app_id String, + device_id String, + session_id String, + event_type String, + event_data String, -- JSON + app_version String, + mosis_version String +) ENGINE = MergeTree() +PARTITION BY toYYYYMM(timestamp) +ORDER BY (app_id, timestamp); +``` + +### Option C: Custom + PostgreSQL + +``` +Raw events → Write to append-only log +Aggregator → Process hourly → Write to PostgreSQL +Cleanup → Delete raw after 24h +``` + +--- + +## Aggregation + +### Pre-computed Metrics + +| Metric | Granularity | Retention | +|--------|-------------|-----------| +| Daily active users | Day | 2 years | +| Event counts | Day | 1 year | +| Crash counts | Day | 1 year | +| Session duration | Day | 90 days | +| Performance percentiles | Day | 90 days | + +### Aggregation Queries + +```sql +-- Daily active users +SELECT + DATE_TRUNC('day', time) as day, + COUNT(DISTINCT device_id) as dau +FROM telemetry_events +WHERE app_id = $1 + AND event_type = 'app_start' + AND time > NOW() - INTERVAL '30 days' +GROUP BY day +ORDER BY day; + +-- Crash rate by version +SELECT + app_version, + COUNT(*) FILTER (WHERE event_type = 'app_crash') as crashes, + COUNT(*) FILTER (WHERE event_type = 'app_start') as starts, + ROUND( + 100.0 * COUNT(*) FILTER (WHERE event_type = 'app_crash') / + NULLIF(COUNT(*) FILTER (WHERE event_type = 'app_start'), 0), + 2 + ) as crash_rate +FROM telemetry_events +WHERE app_id = $1 + AND time > NOW() - INTERVAL '7 days' +GROUP BY app_version; +``` + +--- + +## Crash Grouping + +### Stack Trace Fingerprinting + +```go +func fingerprintCrash(crash CrashReport) string { + // Normalize stack trace + normalized := normalizeStackTrace(crash.StackTrace) + + // Hash key components + key := fmt.Sprintf("%s:%s:%s", + crash.CrashType, + crash.Message, + normalized, + ) + + return sha256(key)[:16] +} + +func normalizeStackTrace(stack string) string { + // Remove line numbers (they change with code updates) + // Remove memory addresses + // Keep function names and file names + re := regexp.MustCompile(`:\d+:`) + return re.ReplaceAllString(stack, ":?:") +} +``` + +### Crash Groups Table + +```sql +CREATE TABLE crash_groups ( + id UUID PRIMARY KEY, + app_id TEXT NOT NULL, + fingerprint TEXT NOT NULL, + crash_type TEXT NOT NULL, + message TEXT, + sample_stack_trace TEXT, + first_seen TIMESTAMPTZ NOT NULL, + last_seen TIMESTAMPTZ NOT NULL, + occurrence_count INT DEFAULT 1, + affected_versions TEXT[], + status TEXT DEFAULT 'open', -- open, resolved, ignored + UNIQUE(app_id, fingerprint) +); +``` + +--- + +## Developer Dashboard + +### Metrics View + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Analytics - My Calculator │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Date Range: [Last 30 days ▼] │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Daily Users │ │ Crashes │ │ Crash-free │ │ +│ │ 1,234 │ │ 23 │ │ 98.1% │ │ +│ │ ▲ +12% │ │ ▼ -45% │ │ ▲ +2% │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Daily Active Users │ │ +│ │ [Line chart showing DAU over time] │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Version Distribution │ │ +│ │ [Pie chart: v1.2.0: 60%, v1.1.0: 30%, v1.0.0: 10%]│ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Crashes View + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Crashes - My Calculator │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Filter: [All versions ▼] [Open ▼] │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ ● attempt to index nil value 'user' │ │ +│ │ lua_error • 156 occurrences • v1.2.0 │ │ +│ │ First: Jan 10 • Last: Jan 15 │ │ +│ │ [View] │ │ +│ ├──────────────────────────────────────────────────────┤ │ +│ │ ● memory limit exceeded │ │ +│ │ sandbox_error • 23 occurrences • v1.1.0, v1.2.0 │ │ +│ │ First: Jan 5 • Last: Jan 14 │ │ +│ │ [View] │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## API Endpoints + +```yaml +# Ingestion (from devices) +POST /v1/telemetry/events: + auth: device_token or api_key + body: { app_id, device_id, events[] } + response: { received: number } + +POST /v1/telemetry/crash: + auth: device_token or api_key + body: { app_id, device_id, crash } + response: { id: string } + +# Dashboard (for developers) +GET /v1/apps/:id/analytics/overview: + auth: required + query: { start_date, end_date } + response: { dau, crashes, crash_free_rate, ... } + +GET /v1/apps/:id/analytics/events: + auth: required + query: { start_date, end_date, event_type } + response: { data: [{ date, count, unique_devices }] } + +GET /v1/apps/:id/crashes: + auth: required + query: { version, status, page, limit } + response: { crashes: CrashGroup[], total } + +GET /v1/apps/:id/crashes/:fingerprint: + auth: required + response: { crash_group, recent_occurrences[] } + +PATCH /v1/apps/:id/crashes/:fingerprint: + auth: required + body: { status: 'resolved' | 'ignored' } + response: { crash_group } +``` + +--- + +## Data Retention + +| Data Type | Retention | Reason | +|-----------|-----------|--------| +| Raw events | 7 days | Debugging | +| Daily aggregates | 2 years | Trends | +| Crash reports | 90 days | Investigation | +| Crash groups | Forever | Issue tracking | + +### Cleanup Job + +```sql +-- Run daily +DELETE FROM telemetry_events +WHERE time < NOW() - INTERVAL '7 days'; + +DELETE FROM crash_reports +WHERE timestamp < NOW() - INTERVAL '90 days'; +``` + +--- + +## Privacy Controls + +### User Settings + +``` +Settings > Privacy > Analytics +├── [✓] Send crash reports (helps developers fix bugs) +├── [ ] Send usage analytics (how you use apps) +└── [Request Data Deletion] +``` + +### GDPR Endpoints + +```yaml +# User requests their data +GET /v1/privacy/export: + auth: user_token + response: { download_url } # JSON export of all data + +# User requests deletion +DELETE /v1/privacy/data: + auth: user_token + response: { status: 'scheduled' } # Delete within 30 days +``` + +--- + +## Deliverables + +- [ ] Event schema specification +- [ ] Client-side SDK for batching +- [ ] Ingestion API endpoints +- [ ] Storage setup (TimescaleDB or ClickHouse) +- [ ] Aggregation jobs +- [ ] Crash grouping logic +- [ ] Developer dashboard +- [ ] Privacy controls +- [ ] Data retention automation +- [ ] GDPR export/delete + +--- + +## Open Questions + +1. Real-time crash alerts (email/Slack)? +2. Sampling for high-volume apps? +3. Custom events API for developers? +4. Benchmarks/comparisons with similar apps? + +--- + +## References + +- [GDPR Requirements](https://gdpr.eu/) +- [TimescaleDB Best Practices](https://docs.timescale.com/timescaledb/latest/) +- [Sentry Crash Grouping](https://docs.sentry.io/product/data-management-settings/event-grouping/) diff --git a/DEV_PORTAL_M09_REVIEW.md b/DEV_PORTAL_M09_REVIEW.md new file mode 100644 index 0000000..df811f8 --- /dev/null +++ b/DEV_PORTAL_M09_REVIEW.md @@ -0,0 +1,474 @@ +# Milestone 9: App Review System + +**Status**: Planning +**Goal**: Automated and manual review process for app submissions. + +--- + +## Overview + +The review system ensures apps meet quality and security standards before publication. Balances automation with manual review for edge cases. + +--- + +## Review Pipeline + +``` +┌─────────┐ ┌───────────┐ ┌─────────┐ ┌──────────┐ ┌───────────┐ +│ Submit │──►│ Automated │──►│ Manual │──►│ Approved │──►│ Published │ +│ │ │ Checks │ │ Review │ │ │ │ │ +└─────────┘ └───────────┘ └─────────┘ └──────────┘ └───────────┘ + │ │ + ▼ ▼ + ┌─────────┐ ┌─────────┐ + │ Failed │ │Rejected │ + │(auto-fix)│ │(feedback)│ + └─────────┘ └─────────┘ +``` + +--- + +## Automated Checks + +### Tier 1: Package Validation (Blocking) + +| Check | Description | Action on Fail | +|-------|-------------|----------------| +| Valid ZIP | Package is valid ZIP format | Reject | +| Manifest exists | manifest.json at root | Reject | +| Manifest valid | JSON parses, required fields | Reject | +| Signature valid | Package signed with registered key | Reject | +| Entry exists | Entry point file exists | Reject | +| Size limits | Under max package size | Reject | +| No forbidden files | No .exe, .dll, etc. | Reject | + +### Tier 2: Content Validation (Blocking) + +| Check | Description | Action on Fail | +|-------|-------------|----------------| +| RML valid | All RML files parse | Reject | +| RCSS valid | All RCSS files parse | Reject | +| Lua syntax | All Lua files parse | Reject | +| Icons valid | Icons are valid images | Reject | +| Icon sizes | Required icon sizes present | Reject | +| Path safety | No path traversal attempts | Reject | + +### Tier 3: Security Analysis (Warning/Flag) + +| Check | Description | Action on Fail | +|-------|-------------|----------------| +| Dangerous patterns | Known malicious Lua patterns | Flag for review | +| Excessive permissions | Unusual permission combos | Flag for review | +| Obfuscated code | Heavily obfuscated Lua | Flag for review | +| External URLs | Hardcoded external URLs | Flag for review | +| Large assets | Unusually large files | Warning | + +### Tier 4: Quality Checks (Warning) + +| Check | Description | Action on Fail | +|-------|-------------|----------------| +| Description length | Meaningful description | Warning | +| Release notes | Non-empty release notes | Warning | +| Icon quality | Not placeholder/blank | Warning | +| Localization | Locale files complete | Warning | + +--- + +## Implementation + +### Validation Worker + +```go +type ValidationResult struct { + Passed bool + Errors []ValidationError + Warnings []ValidationWarning + Flags []ReviewFlag +} + +type ValidationError struct { + Code string + Message string + File string + Line int +} + +func ValidatePackage(packagePath string) ValidationResult { + result := ValidationResult{Passed: true} + + // Tier 1: Package validation + if err := validateZip(packagePath); err != nil { + result.AddError("INVALID_ZIP", err.Error()) + return result + } + + manifest, err := extractManifest(packagePath) + if err != nil { + result.AddError("INVALID_MANIFEST", err.Error()) + return result + } + + if err := validateSignature(packagePath, manifest); err != nil { + result.AddError("INVALID_SIGNATURE", err.Error()) + return result + } + + // Tier 2: Content validation + files, _ := listFiles(packagePath) + for _, file := range files { + switch filepath.Ext(file) { + case ".rml": + if err := validateRML(file); err != nil { + result.AddError("INVALID_RML", err.Error(), file) + } + case ".rcss": + if err := validateRCSS(file); err != nil { + result.AddError("INVALID_RCSS", err.Error(), file) + } + case ".lua": + if err := validateLua(file); err != nil { + result.AddError("INVALID_LUA", err.Error(), file) + } + if flags := analyzeLuaSecurity(file); len(flags) > 0 { + result.Flags = append(result.Flags, flags...) + } + } + } + + // Tier 3: Security analysis + if hasDangerousPatterns(files) { + result.AddFlag("DANGEROUS_PATTERNS", "Code contains suspicious patterns") + } + + // Tier 4: Quality checks + if len(manifest.Description) < 10 { + result.AddWarning("SHORT_DESCRIPTION", "Description is very short") + } + + result.Passed = len(result.Errors) == 0 + return result +} +``` + +### Dangerous Pattern Detection + +```go +var dangerousPatterns = []struct { + Pattern *regexp.Regexp + Reason string +}{ + { + regexp.MustCompile(`loadstring\s*\(`), + "Dynamic code execution", + }, + { + regexp.MustCompile(`debug\s*\.\s*\w+`), + "Debug library usage", + }, + { + regexp.MustCompile(`os\s*\.\s*execute`), + "OS command execution", + }, + { + regexp.MustCompile(`io\s*\.\s*\w+`), + "Direct I/O operations", + }, + { + regexp.MustCompile(`ffi\s*\.\s*\w+`), + "FFI usage", + }, + { + regexp.MustCompile(`package\s*\.\s*loadlib`), + "Native library loading", + }, +} + +func analyzeLuaSecurity(content string) []ReviewFlag { + var flags []ReviewFlag + for _, dp := range dangerousPatterns { + if dp.Pattern.MatchString(content) { + flags = append(flags, ReviewFlag{ + Type: "SECURITY", + Reason: dp.Reason, + }) + } + } + return flags +} +``` + +--- + +## Manual Review + +### When Required + +| Trigger | Reason | +|---------|--------| +| New developer | First app submission | +| Dangerous permissions | camera, microphone, contacts, location | +| Security flags | Automated checks flagged concerns | +| User reports | Existing app reported | +| Appeal | Developer contests rejection | + +### Review Queue UI + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Review Queue [14 pending] │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Filter: [All ▼] [Flagged first ▼] │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ 🚩 Weather Pro v2.0.0 │ │ +│ │ com.newdev.weather • New developer │ │ +│ │ Permissions: location, network │ │ +│ │ Flags: First submission │ │ +│ │ Submitted: 2 hours ago │ │ +│ │ [Review] [Auto-approve]│ │ +│ ├──────────────────────────────────────────────────────┤ │ +│ │ 🚩 Photo Editor v1.5.0 │ │ +│ │ com.trusted.photos • Verified developer │ │ +│ │ Permissions: camera, storage │ │ +│ │ Flags: DANGEROUS_PATTERNS (1) │ │ +│ │ Submitted: 5 hours ago │ │ +│ │ [Review] [Auto-approve]│ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Review Detail View + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Review: Weather Pro v2.0.0 │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Package: com.newdev.weather │ +│ Developer: newdev@example.com (New - first app) │ +│ Submitted: Jan 15, 2024 10:30 AM │ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Automated Checks │ │ +│ │ ✓ Package validation passed │ │ +│ │ ✓ Content validation passed │ │ +│ │ ⚠ Warning: SHORT_DESCRIPTION │ │ +│ │ 🚩 Flag: First submission from new developer │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ Permissions Requested: │ +│ • location (fine) - Used for weather forecasts │ +│ • network - Required for API calls │ +│ │ +│ Files: [Expand to browse] │ +│ ├── manifest.json │ +│ ├── assets/ │ +│ │ ├── main.rml │ +│ │ └── scripts/ │ +│ │ └── weather.lua [View source] │ +│ └── icons/ │ +│ │ +│ Reviewer Notes: │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ [Approve] [Reject with feedback] [Request changes] │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Review States + +``` + ┌─────────┐ + │ Draft │ + └────┬────┘ + │ submit + ▼ + ┌─────────┐ + ┌───────│Uploaded │ + │ └────┬────┘ + │ │ validation + │ ▼ + │ ┌──────────┐ + failed │ │Validating│ + │ └────┬─────┘ + │ │ + │ ┌───────┴───────┐ + │ │ │ + ▼ ▼ ▼ + ┌────────┐ ┌─────────┐ + │ Failed │ │In Review│ + └────────┘ └────┬────┘ + │ + ┌───────────┼───────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌────────┐ ┌──────────┐ + │ Approved │ │Rejected│ │ Changes │ + └────┬─────┘ └────────┘ │ Requested│ + │ └──────────┘ + │ publish + ▼ + ┌──────────┐ + │Published │ + └──────────┘ +``` + +--- + +## Rejection Feedback + +### Feedback Template + +```json +{ + "reason": "SECURITY_CONCERN", + "message": "Your app contains patterns that raise security concerns.", + "details": [ + { + "file": "scripts/main.lua", + "line": 42, + "issue": "Usage of loadstring() is not allowed", + "suggestion": "Use static Lua code instead of dynamic evaluation" + } + ], + "can_resubmit": true, + "appeal_available": true +} +``` + +### Common Rejection Reasons + +| Code | Description | +|------|-------------| +| `SECURITY_CONCERN` | Security issues found | +| `QUALITY_ISSUE` | Doesn't meet quality standards | +| `POLICY_VIOLATION` | Violates content policy | +| `METADATA_ISSUE` | Incorrect/misleading metadata | +| `PERMISSION_ABUSE` | Unnecessary permissions | +| `COPYRIGHT` | Copyright/trademark issues | + +--- + +## Appeal Process + +``` +1. Developer receives rejection +2. Developer clicks "Appeal" (within 14 days) +3. Provides justification +4. Different reviewer examines +5. Final decision (approve/uphold rejection) +``` + +--- + +## Review SLA + +| Submission Type | Target Time | +|-----------------|-------------| +| Auto-approved | Instant | +| New developer | 24 hours | +| Flagged | 48 hours | +| Appeal | 72 hours | + +--- + +## API Endpoints + +```yaml +# Developer endpoints +POST /v1/apps/:id/versions/:vid/submit: + summary: Submit for review + response: { version, estimated_review_time } + +GET /v1/apps/:id/versions/:vid/review-status: + summary: Get review status + response: { status, feedback?, estimated_completion } + +POST /v1/apps/:id/versions/:vid/appeal: + summary: Appeal rejection + body: { justification } + response: { appeal_id, status } + +# Internal review endpoints (admin only) +GET /v1/admin/review-queue: + summary: Get pending reviews + response: { items[], total } + +GET /v1/admin/review/:version_id: + summary: Get review details + response: { version, validation_result, flags } + +POST /v1/admin/review/:version_id/approve: + summary: Approve version + body: { notes? } + response: { version } + +POST /v1/admin/review/:version_id/reject: + summary: Reject version + body: { reason, message, details[] } + response: { version } +``` + +--- + +## Metrics + +### Review Performance + +| Metric | Target | +|--------|--------| +| Auto-approval rate | > 70% | +| Average review time | < 24 hours | +| Rejection rate | < 20% | +| Appeal overturn rate | < 10% | + +### Dashboard + +```sql +-- Review stats +SELECT + DATE_TRUNC('week', submitted_at) as week, + COUNT(*) as submissions, + COUNT(*) FILTER (WHERE status = 'published') as approved, + COUNT(*) FILTER (WHERE status = 'rejected') as rejected, + AVG(EXTRACT(EPOCH FROM (reviewed_at - submitted_at))/3600) as avg_hours +FROM app_versions +WHERE submitted_at > NOW() - INTERVAL '30 days' +GROUP BY week; +``` + +--- + +## Deliverables + +- [ ] Validation worker implementation +- [ ] Dangerous pattern database +- [ ] Review queue UI +- [ ] Reviewer tools +- [ ] Rejection feedback system +- [ ] Appeal workflow +- [ ] Review metrics dashboard +- [ ] SLA monitoring + +--- + +## Open Questions + +1. Automated approval for trusted developers? +2. Community moderators? +3. Content policy document? +4. Rate limiting resubmissions? + +--- + +## References + +- [Apple App Review Guidelines](https://developer.apple.com/app-store/review/guidelines/) +- [Google Play Policy](https://play.google.com/about/developer-content-policy/) diff --git a/DEV_PORTAL_M10_DEVICE.md b/DEV_PORTAL_M10_DEVICE.md new file mode 100644 index 0000000..c5e27e3 --- /dev/null +++ b/DEV_PORTAL_M10_DEVICE.md @@ -0,0 +1,603 @@ +# Milestone 10: Device-Side App Management + +**Status**: Planning +**Goal**: Install, update, and manage apps on Mosis devices. + +--- + +## Overview + +Device-side app management handles the full lifecycle of apps on user devices: discovery, installation, updates, and removal. + +--- + +## Components + +``` +┌─────────────────────────────────────────────────────────────┐ +│ MosisService │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ +│ │ AppManager │ │ UpdateService │ │ AppStore UI │ │ +│ │ (C++ class) │ │ (Background) │ │ (System App) │ │ +│ └───────────────┘ └───────────────┘ └───────────────┘ │ +│ │ │ │ │ +│ └───────────────────┼───────────────────┘ │ +│ │ │ +│ ┌────────┴────────┐ │ +│ │ LuaSandboxManager│ │ +│ └─────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Storage Layout + +``` +/data/mosis/ +├── config/ +│ ├── device.json # Device ID, settings +│ └── apps.json # Installed apps registry +├── apps/ +│ └── {package_id}/ +│ ├── package/ # Extracted app files +│ │ ├── manifest.json +│ │ └── assets/ +│ ├── data/ # App data (VirtualFS) +│ ├── cache/ # App cache +│ └── db/ # SQLite databases +├── downloads/ # Temp download location +└── backups/ # App data backups (before update) +``` + +--- + +## AppManager Class + +### Interface + +```cpp +namespace mosis { + +struct InstalledApp { + std::string package_id; + std::string name; + std::string version_name; + int version_code; + std::string install_path; + std::vector permissions; + std::chrono::system_clock::time_point installed_at; + std::chrono::system_clock::time_point updated_at; + int64_t package_size; + int64_t data_size; +}; + +struct InstallProgress { + enum class Stage { + Downloading, + Verifying, + Extracting, + Registering, + Complete, + Failed + }; + Stage stage; + float progress; // 0.0 - 1.0 + std::string error; +}; + +using ProgressCallback = std::function; + +class AppManager { +public: + explicit AppManager(const std::string& data_root); + ~AppManager(); + + // Installation + bool Install(const std::string& package_url, + const std::string& signature, + ProgressCallback callback); + bool InstallFromFile(const std::string& package_path, + ProgressCallback callback); + + // Uninstallation + bool Uninstall(const std::string& package_id, bool keep_data = false); + + // Updates + bool Update(const std::string& package_id, + const std::string& package_url, + const std::string& signature, + ProgressCallback callback); + + // Queries + std::vector GetInstalledApps() const; + std::optional GetApp(const std::string& package_id) const; + bool IsInstalled(const std::string& package_id) const; + + // Data management + int64_t GetAppDataSize(const std::string& package_id) const; + bool ClearAppData(const std::string& package_id); + bool ClearAppCache(const std::string& package_id); + bool BackupAppData(const std::string& package_id); + bool RestoreAppData(const std::string& package_id); + + // Integration with sandbox + void SetSandboxManager(LuaSandboxManager* manager); + +private: + std::string m_data_root; + LuaSandboxManager* m_sandbox_manager = nullptr; + mutable std::mutex m_mutex; + std::map m_installed_apps; + + bool VerifyPackage(const std::string& path, const std::string& signature); + bool ExtractPackage(const std::string& path, const std::string& dest); + void LoadInstalledApps(); + void SaveInstalledApps(); +}; + +} // namespace mosis +``` + +### Implementation + +```cpp +bool AppManager::Install(const std::string& package_url, + const std::string& signature, + ProgressCallback callback) { + callback({InstallProgress::Stage::Downloading, 0.0f, ""}); + + // Download package + std::string download_path = m_data_root + "/downloads/" + GenerateUUID(); + if (!DownloadFile(package_url, download_path, [&](float p) { + callback({InstallProgress::Stage::Downloading, p, ""}); + })) { + callback({InstallProgress::Stage::Failed, 0.0f, "Download failed"}); + return false; + } + + callback({InstallProgress::Stage::Verifying, 0.0f, ""}); + + // Verify signature + if (!VerifyPackage(download_path, signature)) { + std::filesystem::remove(download_path); + callback({InstallProgress::Stage::Failed, 0.0f, "Signature verification failed"}); + return false; + } + + // Extract manifest to get package_id + auto manifest = ExtractManifest(download_path); + if (!manifest) { + std::filesystem::remove(download_path); + callback({InstallProgress::Stage::Failed, 0.0f, "Invalid manifest"}); + return false; + } + + callback({InstallProgress::Stage::Extracting, 0.0f, ""}); + + // Check if already installed (update path) + std::string install_path = m_data_root + "/apps/" + manifest->package_id; + if (std::filesystem::exists(install_path + "/package")) { + // Backup existing data + BackupAppData(manifest->package_id); + // Remove old package + std::filesystem::remove_all(install_path + "/package"); + } + + // Extract package + std::filesystem::create_directories(install_path + "/package"); + if (!ExtractPackage(download_path, install_path + "/package")) { + callback({InstallProgress::Stage::Failed, 0.0f, "Extraction failed"}); + return false; + } + + // Clean up download + std::filesystem::remove(download_path); + + callback({InstallProgress::Stage::Registering, 0.0f, ""}); + + // Create data directories + std::filesystem::create_directories(install_path + "/data"); + std::filesystem::create_directories(install_path + "/cache"); + std::filesystem::create_directories(install_path + "/db"); + + // Register app + InstalledApp app{ + .package_id = manifest->package_id, + .name = manifest->name, + .version_name = manifest->version, + .version_code = manifest->version_code, + .install_path = install_path, + .permissions = manifest->permissions, + .installed_at = std::chrono::system_clock::now(), + .updated_at = std::chrono::system_clock::now(), + .package_size = std::filesystem::file_size(download_path) + }; + + { + std::lock_guard lock(m_mutex); + m_installed_apps[manifest->package_id] = app; + SaveInstalledApps(); + } + + callback({InstallProgress::Stage::Complete, 1.0f, ""}); + return true; +} + +bool AppManager::Uninstall(const std::string& package_id, bool keep_data) { + std::lock_guard lock(m_mutex); + + auto it = m_installed_apps.find(package_id); + if (it == m_installed_apps.end()) { + return false; + } + + // Stop app if running + if (m_sandbox_manager && m_sandbox_manager->IsAppRunning(package_id)) { + m_sandbox_manager->StopApp(package_id); + } + + // Remove files + std::string install_path = it->second.install_path; + std::filesystem::remove_all(install_path + "/package"); + std::filesystem::remove_all(install_path + "/cache"); + + if (!keep_data) { + std::filesystem::remove_all(install_path + "/data"); + std::filesystem::remove_all(install_path + "/db"); + std::filesystem::remove_all(install_path); + } + + // Unregister + m_installed_apps.erase(it); + SaveInstalledApps(); + + return true; +} +``` + +--- + +## Update Service + +### Background Update Checker + +```cpp +class UpdateService { +public: + UpdateService(AppManager* app_manager, const std::string& api_base); + + // Start background checking + void Start(std::chrono::hours interval = std::chrono::hours(24)); + void Stop(); + + // Manual check + std::vector CheckForUpdates(); + + // Download and install update + bool ApplyUpdate(const std::string& package_id, ProgressCallback callback); + + // Settings + void SetAutoUpdate(bool enabled); + void SetWifiOnly(bool wifi_only); + +private: + void CheckLoop(); + + AppManager* m_app_manager; + std::string m_api_base; + std::thread m_check_thread; + std::atomic m_running{false}; + bool m_auto_update = false; + bool m_wifi_only = true; +}; +``` + +### Update Check Flow + +``` +1. Get list of installed apps +2. Call API: GET /store/apps/updates?packages=com.a,com.b,com.c +3. API returns list of available updates +4. If auto-update enabled and on WiFi: + - Download and install in background + - Notify user of completed updates +5. If manual: + - Show notification with update count + - User opens App Store to review +``` + +--- + +## App Store System App + +### UI Screens + +``` +Home +├── Featured apps +├── Categories +├── Search +└── My Apps + ├── Installed + ├── Updates available + └── Previously installed + +App Detail +├── Icon, name, developer +├── Screenshots +├── Description +├── Permissions +├── Reviews (future) +└── [Install] / [Update] / [Open] + +Settings +├── Auto-update (WiFi only) +├── Storage usage +└── Clear all caches +``` + +### RML Structure + +```xml + + + + + +