omar/MosisService
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

finalize Go + SQLite stack decisions for Synology NAS deployment

Browse Source
This commit is contained in:
omigamedev
2026-01-18 18:52:08 +01:00
parent 30a7146929
commit 416c447ad8
2 changed files with 233 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
DEV_PORTAL_M02_WEB_STACK.md
View File

@@ -1,8 +1,44 @@
# Milestone 2: Web Stack Selection # Milestone 2: Web Stack Selection
**Status**: Planning **Status**: Decided
**Goal**: Choose backend technologies for the developer portal and app store API. **Goal**: Choose backend technologies for the developer portal and app store API.
## Decision
**Go** with the following stack:
```
Language: Go 1.22+
Framework: Chi (lightweight, idiomatic)
Database: SQLite via modernc.org/sqlite (pure Go, no CGO)
Migrations: golang-migrate
Validation: go-playground/validator
Auth: Custom JWT + OAuth2
Deployment: Single Docker container on Synology NAS
```
### Rationale
1. **NAS-friendly** - Tiny Docker image (~15MB), low RAM (~30-50MB)
2. **Cross-compilation** - Easy build for ARM64 or AMD64 Synology models
3. **Pure Go SQLite** - `modernc.org/sqlite` requires no CGO, cross-compiles easily
4. **Single container** - Go binary + SQLite + Litestream in one image
5. **Standard library** - HTTP, JSON, crypto all built-in
### Target Deployment
```
Synology NAS (Docker)
├── mosis-portal container (~15MB image)
│ ├── Go binary
│ ├── SQLite database (WAL mode)
│ └── Litestream backup
└── Volumes
├── /volume1/mosis/data/portal.db
├── /volume1/mosis/backups/
└── /volume1/mosis/packages/
```
--- ---
## Overview ## Overview
@@ -308,12 +344,10 @@ Based on:
## Deliverables ## Deliverables
- [ ] Prototype API in Go - [x] Final selection with rationale
- [ ] Prototype API in Node.js (if needed)
- [ ] Benchmark comparison document
- [ ] Final selection with rationale
- [ ] Project structure template - [ ] Project structure template
- [ ] Development environment setup guide - [ ] Development environment setup guide
- [ ] Base Go project scaffolding
--- ---

279
DEV_PORTAL_M03_DATABASE.md
View File

@@ -1,8 +1,61 @@
# Milestone 3: Database Selection # Milestone 3: Database Selection
**Status**: Planning **Status**: Decided
**Goal**: Choose database for developer accounts, app metadata, and analytics. **Goal**: Choose database for developer accounts, app metadata, and analytics.
## Decision
**SQLite + Litestream** for self-hosted deployment on Synology NAS.
```
Database: SQLite 3.x (WAL mode)
Driver: modernc.org/sqlite (pure Go, no CGO)
Backup: Litestream continuous replication
Storage: Synology volume (/volume1/mosis/)
```
### Rationale
1. **Single container** - No separate database service needed
2. **Minimal resources** - ~50MB RAM, perfect for NAS
3. **Zero ops** - No connection pooling, no tuning
4. **Continuous backup** - Litestream replicates to local storage
5. **Point-in-time recovery** - Restore to any moment
6. **Sufficient scale** - Handles 1000s of developers easily
### Architecture
```
┌─────────────────────────────────────────┐
│ Synology NAS │
│ ┌─────────────────────────────────┐ │
│ │ mosis-portal container │ │
│ │ ├── Go binary │ │
│ │ ├── SQLite (portal.db) │ │
│ │ └── Litestream │ │
│ └──────────────┬──────────────────┘ │
│ │ │
│ ┌──────────────▼──────────────────┐ │
│ │ /volume1/mosis/ │ │
│ │ ├── data/portal.db │ │
│ │ ├── data/portal.db-wal │ │
│ │ ├── backups/ (litestream) │ │
│ │ └── packages/ (app uploads) │ │
│ └─────────────────────────────────┘ │
└─────────────────────────────────────────┘
```
### Litestream Configuration
```yaml
dbs:
- path: /data/portal.db
replicas:
- type: file
path: /backups/portal
retention: 720h # 30 days
```
--- ---
## Overview ## Overview
@@ -161,127 +214,140 @@ Redis → Caching, Sessions
--- ---
## Schema Design ## Schema Design (SQLite)
### Core Tables ### Core Tables
```sql ```sql
-- Developers -- Developers
CREATE TABLE developers ( CREATE TABLE developers (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id TEXT PRIMARY KEY, -- UUID as text
email VARCHAR(255) UNIQUE NOT NULL, email TEXT UNIQUE NOT NULL,
name VARCHAR(100) NOT NULL, name TEXT NOT NULL,
password_hash VARCHAR(255), password_hash TEXT,
oauth_provider VARCHAR(50), oauth_provider TEXT,
oauth_id VARCHAR(255), oauth_id TEXT,
verified BOOLEAN DEFAULT FALSE, verified INTEGER DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW(), created_at TEXT DEFAULT (datetime('now')),
updated_at TIMESTAMPTZ DEFAULT NOW() updated_at TEXT DEFAULT (datetime('now'))
); );
-- API Keys -- API Keys
CREATE TABLE api_keys ( CREATE TABLE api_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id TEXT PRIMARY KEY,
developer_id UUID REFERENCES developers(id) ON DELETE CASCADE, developer_id TEXT NOT NULL REFERENCES developers(id) ON DELETE CASCADE,
name VARCHAR(100) NOT NULL, name TEXT NOT NULL,
key_hash VARCHAR(255) NOT NULL, key_hash TEXT NOT NULL,
key_prefix VARCHAR(10) NOT NULL, -- For display: "mk_abc..." key_prefix TEXT NOT NULL, -- For display: "mk_abc..."
permissions JSONB DEFAULT '[]', permissions TEXT DEFAULT '[]', -- JSON array
last_used_at TIMESTAMPTZ, last_used_at TEXT,
expires_at TIMESTAMPTZ, expires_at TEXT,
created_at TIMESTAMPTZ DEFAULT NOW() created_at TEXT DEFAULT (datetime('now'))
); );
-- Apps -- Apps
CREATE TABLE apps ( CREATE TABLE apps (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id TEXT PRIMARY KEY,
developer_id UUID REFERENCES developers(id) ON DELETE CASCADE, developer_id TEXT NOT NULL REFERENCES developers(id) ON DELETE CASCADE,
package_id VARCHAR(255) UNIQUE NOT NULL, -- com.dev.app package_id TEXT UNIQUE NOT NULL, -- com.dev.app
name VARCHAR(100) NOT NULL, name TEXT NOT NULL,
description TEXT, description TEXT,
category VARCHAR(50), category TEXT,
tags VARCHAR(50)[] DEFAULT '{}', tags TEXT DEFAULT '[]', -- JSON array
status VARCHAR(20) DEFAULT 'draft', -- draft, published, suspended status TEXT DEFAULT 'draft', -- draft, published, suspended
created_at TIMESTAMPTZ DEFAULT NOW(), created_at TEXT DEFAULT (datetime('now')),
updated_at TIMESTAMPTZ DEFAULT NOW() updated_at TEXT DEFAULT (datetime('now'))
); );
-- App Versions -- App Versions
CREATE TABLE app_versions ( CREATE TABLE app_versions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id TEXT PRIMARY KEY,
app_id UUID REFERENCES apps(id) ON DELETE CASCADE, app_id TEXT NOT NULL REFERENCES apps(id) ON DELETE CASCADE,
version_code INTEGER NOT NULL, version_code INTEGER NOT NULL,
version_name VARCHAR(20) NOT NULL, version_name TEXT NOT NULL,
package_url TEXT NOT NULL, package_url TEXT NOT NULL,
package_size BIGINT NOT NULL, package_size INTEGER NOT NULL,
signature VARCHAR(512) NOT NULL, signature TEXT NOT NULL,
permissions JSONB DEFAULT '[]', permissions TEXT DEFAULT '[]', -- JSON array
min_mosis_version VARCHAR(20), min_mosis_version TEXT,
release_notes TEXT, release_notes TEXT,
status VARCHAR(20) DEFAULT 'draft', -- draft, review, approved, published, rejected status TEXT DEFAULT 'draft', -- draft, review, approved, published, rejected
review_notes TEXT, review_notes TEXT,
published_at TIMESTAMPTZ, published_at TEXT,
created_at TIMESTAMPTZ DEFAULT NOW(), created_at TEXT DEFAULT (datetime('now')),
UNIQUE(app_id, version_code) UNIQUE(app_id, version_code)
); );
-- Developer Signing Keys -- Developer Signing Keys
CREATE TABLE signing_keys ( CREATE TABLE signing_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id TEXT PRIMARY KEY,
developer_id UUID REFERENCES developers(id) ON DELETE CASCADE, developer_id TEXT NOT NULL REFERENCES developers(id) ON DELETE CASCADE,
name VARCHAR(100) NOT NULL, name TEXT NOT NULL,
public_key TEXT NOT NULL, public_key TEXT NOT NULL,
fingerprint VARCHAR(64) NOT NULL, fingerprint TEXT NOT NULL,
is_active BOOLEAN DEFAULT TRUE, is_active INTEGER DEFAULT 1,
created_at TIMESTAMPTZ DEFAULT NOW() created_at TEXT DEFAULT (datetime('now'))
); );
``` ```
### Telemetry Tables (if using PostgreSQL) ### Telemetry Tables
```sql ```sql
-- Telemetry Events (consider partitioning by time) -- Telemetry Events (append-only, partition by month via separate tables)
CREATE TABLE telemetry_events ( CREATE TABLE telemetry_events (
id BIGSERIAL, id INTEGER PRIMARY KEY AUTOINCREMENT,
app_id UUID NOT NULL, app_id TEXT NOT NULL,
device_id VARCHAR(64) NOT NULL, -- Hashed device_id TEXT NOT NULL, -- Hashed for privacy
event_type VARCHAR(50) NOT NULL, event_type TEXT NOT NULL,
event_data JSONB, event_data TEXT, -- JSON string
mosis_version VARCHAR(20), mosis_version TEXT,
timestamp TIMESTAMPTZ NOT NULL, timestamp TEXT NOT NULL -- ISO8601 format
PRIMARY KEY (timestamp, id) );
) PARTITION BY RANGE (timestamp);
-- Create monthly partitions
CREATE TABLE telemetry_events_2024_01 PARTITION OF telemetry_events
FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');
-- Crash Reports -- Crash Reports
CREATE TABLE crash_reports ( CREATE TABLE crash_reports (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id TEXT PRIMARY KEY,
app_id UUID NOT NULL, app_id TEXT NOT NULL,
app_version VARCHAR(20) NOT NULL, app_version TEXT NOT NULL,
device_id VARCHAR(64) NOT NULL, device_id TEXT NOT NULL,
crash_type VARCHAR(50) NOT NULL, crash_type TEXT NOT NULL,
message TEXT, message TEXT,
stack_trace TEXT, stack_trace TEXT,
context JSONB, context TEXT, -- JSON string
mosis_version VARCHAR(20), mosis_version TEXT,
timestamp TIMESTAMPTZ NOT NULL, timestamp TEXT NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW() created_at TEXT DEFAULT (datetime('now'))
); );
-- Daily aggregates (materialized or computed) -- Daily aggregates (computed by background job)
CREATE TABLE telemetry_daily ( CREATE TABLE telemetry_daily (
app_id UUID NOT NULL, app_id TEXT NOT NULL,
date DATE NOT NULL, date TEXT NOT NULL, -- YYYY-MM-DD
event_type VARCHAR(50) NOT NULL, event_type TEXT NOT NULL,
count BIGINT NOT NULL, count INTEGER NOT NULL,
unique_devices BIGINT NOT NULL, unique_devices INTEGER NOT NULL,
PRIMARY KEY (app_id, date, event_type) PRIMARY KEY (app_id, date, event_type)
); );
-- Audit Logs
CREATE TABLE audit_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
developer_id TEXT,
action TEXT NOT NULL,
resource_type TEXT,
resource_id TEXT,
details TEXT, -- JSON string
ip_address TEXT,
user_agent TEXT,
created_at TEXT DEFAULT (datetime('now'))
);
``` ```
**Note**: For high-volume telemetry, consider:
- Separate SQLite database file for telemetry (isolates write load)
- Monthly table rotation with application-level partitioning
- Aggressive data retention (delete events older than 90 days)
### Indexes ### Indexes
```sql ```sql
@@ -289,16 +355,24 @@ CREATE TABLE telemetry_daily (
CREATE INDEX idx_developers_email ON developers(email); CREATE INDEX idx_developers_email ON developers(email);
CREATE INDEX idx_developers_oauth ON developers(oauth_provider, oauth_id); CREATE INDEX idx_developers_oauth ON developers(oauth_provider, oauth_id);
-- API Keys
CREATE INDEX idx_api_keys_developer ON api_keys(developer_id);
CREATE INDEX idx_api_keys_prefix ON api_keys(key_prefix);
-- Apps -- Apps
CREATE INDEX idx_apps_developer ON apps(developer_id); CREATE INDEX idx_apps_developer ON apps(developer_id);
CREATE INDEX idx_apps_package ON apps(package_id); CREATE INDEX idx_apps_package ON apps(package_id);
CREATE INDEX idx_apps_status ON apps(status); CREATE INDEX idx_apps_status ON apps(status);
CREATE INDEX idx_apps_search ON apps USING gin(to_tsvector('english', name || ' ' || COALESCE(description, ''))); CREATE INDEX idx_apps_name ON apps(name); -- For LIKE searches
-- Versions -- Versions
CREATE INDEX idx_versions_app ON app_versions(app_id); CREATE INDEX idx_versions_app ON app_versions(app_id);
CREATE INDEX idx_versions_status ON app_versions(status); CREATE INDEX idx_versions_status ON app_versions(status);
-- Signing Keys
CREATE INDEX idx_signing_keys_developer ON signing_keys(developer_id);
CREATE INDEX idx_signing_keys_fingerprint ON signing_keys(fingerprint);
-- Telemetry -- Telemetry
CREATE INDEX idx_telemetry_app ON telemetry_events(app_id, timestamp); CREATE INDEX idx_telemetry_app ON telemetry_events(app_id, timestamp);
CREATE INDEX idx_telemetry_type ON telemetry_events(event_type, timestamp); CREATE INDEX idx_telemetry_type ON telemetry_events(event_type, timestamp);
@@ -306,6 +380,41 @@ CREATE INDEX idx_telemetry_type ON telemetry_events(event_type, timestamp);
-- Crashes -- Crashes
CREATE INDEX idx_crashes_app ON crash_reports(app_id, timestamp); CREATE INDEX idx_crashes_app ON crash_reports(app_id, timestamp);
CREATE INDEX idx_crashes_type ON crash_reports(crash_type); CREATE INDEX idx_crashes_type ON crash_reports(crash_type);
-- Audit Logs
CREATE INDEX idx_audit_developer ON audit_logs(developer_id);
CREATE INDEX idx_audit_created ON audit_logs(created_at);
```
**Full-text Search**: For app search, use SQLite FTS5:
```sql
-- Create FTS5 virtual table for app search
CREATE VIRTUAL TABLE apps_fts USING fts5(
name,
description,
tags,
content='apps',
content_rowid='rowid'
);
-- Triggers to keep FTS in sync
CREATE TRIGGER apps_ai AFTER INSERT ON apps BEGIN
INSERT INTO apps_fts(rowid, name, description, tags)
VALUES (NEW.rowid, NEW.name, NEW.description, NEW.tags);
END;
CREATE TRIGGER apps_ad AFTER DELETE ON apps BEGIN
INSERT INTO apps_fts(apps_fts, rowid, name, description, tags)
VALUES ('delete', OLD.rowid, OLD.name, OLD.description, OLD.tags);
END;
CREATE TRIGGER apps_au AFTER UPDATE ON apps BEGIN
INSERT INTO apps_fts(apps_fts, rowid, name, description, tags)
VALUES ('delete', OLD.rowid, OLD.name, OLD.description, OLD.tags);
INSERT INTO apps_fts(rowid, name, description, tags)
VALUES (NEW.rowid, NEW.name, NEW.description, NEW.tags);
END;
``` ```
--- ---
@@ -392,21 +501,21 @@ Redis → Caching, rate limiting
## Deliverables ## Deliverables
- [ ] Final database selection - [x] Final database selection (SQLite + Litestream)
- [ ] Complete schema design - [x] Complete schema design (core + telemetry + FTS5)
- [ ] Migration scripts - [ ] Migration scripts (golang-migrate)
- [ ] Backup/restore procedures - [x] Backup/restore procedures (Litestream to local storage)
- [ ] Connection pooling setup (if PostgreSQL) - [x] ~~Connection pooling setup~~ (not needed for SQLite)
- [ ] Monitoring queries - [ ] Monitoring queries
--- ---
## Open Questions ## Open Questions
1. Expected telemetry volume per day? 1. ~~Expected telemetry volume per day?~~ → Start simple, optimize if needed
2. How long to retain raw telemetry? 2. ~~How long to retain raw telemetry?~~ → 90 days raw, daily aggregates indefinitely
3. Need for real-time analytics vs batch? 3. ~~Need for real-time analytics vs batch?~~ → Batch is sufficient for MVP
4. Multi-region requirements? 4. ~~Multi-region requirements?~~ → Single NAS deployment for now
--- ---