move docs to docs/ folder, merge architecture files, update references

2026-01-19 09:02:11 +01:00
parent 1b34b0e974
commit 010e11cf6b
68 changed files with 1741 additions and 1350 deletions
--- a/docs/DEV_PORTAL_M07_STORAGE.md
+++ b/docs/DEV_PORTAL_M07_STORAGE.md
@@ -0,0 +1,627 @@
+# Milestone 7: CDN & Storage
+
+**Status**: Decided
+**Goal**: Scalable storage for app packages and assets with global distribution.
+
+## Decision
+
+**Local Synology filesystem** as primary storage, with optional Cloudflare R2 for CDN:
+
+```
+Primary:    Synology volume (/volume1/mosis/)
+CDN:        Cloudflare R2 (optional, for global distribution)
+Serving:    Go binary serves files directly (local)
+Backup:     Synology Hyper Backup / rsync
+```
+
+### Rationale
+
+1. **Self-hosted** - All data stays on premises
+2. **Zero cost** - No cloud storage fees
+3. **Simple** - Go binary serves files directly via http.FileServer
+4. **Fast local** - NAS has gigabit+ internal network
+5. **Optional CDN** - Can sync to R2 if global distribution needed
+
+### Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        Synology NAS                              │
+│  ┌────────────────────────────────────────────────────────────┐ │
+│  │                 mosis-portal container                      │ │
+│  │                                                             │ │
+│  │   Go binary ──serves──► /packages/, /assets/               │ │
+│  │        │                                                    │ │
+│  │        └── SQLite (/data/portal.db)                        │ │
+│  └──────────────────────────┬─────────────────────────────────┘ │
+│                              │                                   │
+│  /volume1/mosis/            │ bind mount                        │
+│  ├── data/portal.db         │                                   │
+│  ├── packages/  ◄───────────┘                                   │
+│  │   └── {dev_id}/{app_id}/{version}/package.mosis              │
+│  ├── assets/                                                    │
+│  │   └── {app_id}/icon-{size}.png                              │
+│  └── backups/                                                   │
+│      └── litestream replicas                                    │
+└─────────────────────────────────────────────────────────────────┘
+                              │
+                    (optional sync)
+                              ▼
+                  ┌──────────────────┐
+                  │  Cloudflare R2   │
+                  │  (CDN for global │
+                  │   distribution)  │
+                  └──────────────────┘
+```
+
+---
+
+## Overview
+
+Storage handles app packages, icons, screenshots, and serves downloads to devices. For self-hosted Synology NAS deployment, local filesystem is the primary storage.
+
+---
+
+## Requirements
+
+### Functional
+
+- Store app packages (.mosis files)
+- Store icons (multiple sizes)
+- Store screenshots (optional)
+- Serve downloads globally
+- Support presigned upload URLs
+- Version retention (keep old versions)
+
+### Non-Functional
+
+| Requirement | Target |
+|-------------|--------|
+| Upload speed | < 30s for 50MB |
+| Download latency | < 100ms (p50) |
+| Availability | 99.9% |
+| Durability | 99.999999999% (11 nines) |
+| Cost | Minimize egress fees |
+
+---
+
+## Storage Structure
+
+```
+/packages/
+  /{developer_id}/
+    /{app_id}/
+      /{version_code}/
+        package.mosis
+        manifest.json (extracted for quick access)
+
+/assets/
+  /{app_id}/
+    icon-32.png
+    icon-64.png
+    icon-128.png
+    screenshots/
+      1.png
+      2.png
+      3.png
+
+/temp/
+  /{upload_id}/
+    package.mosis (pending validation)
+```
+
+---
+
+## Options Analysis
+
+### Option A: Cloudflare R2
+
+```
+Storage:    Object storage (S3-compatible)
+CDN:        Cloudflare network (automatic)
+Egress:     FREE (zero egress fees)
+Pricing:    $0.015/GB storage, $4.50/million requests
+```
+
+| Pros | Cons |
+|------|------|
+| No egress fees | Newer service |
+| Global CDN included | Fewer regions than S3 |
+| S3-compatible API | Less tooling |
+| Workers integration | |
+
+#### Cost Estimate (10K apps, 100GB)
+
+| Component | Monthly Cost |
+|-----------|--------------|
+| Storage (100GB) | $1.50 |
+| Requests (1M) | $4.50 |
+| Egress | $0 |
+| **Total** | **~$6** |
+
+---
+
+### Option B: AWS S3 + CloudFront
+
+```
+Storage:    S3 Standard
+CDN:        CloudFront
+Egress:     $0.085-0.12/GB (varies by region)
+Pricing:    $0.023/GB storage
+```
+
+| Pros | Cons |
+|------|------|
+| Most mature | Egress costs add up |
+| Best tooling | Complex pricing |
+| All regions | Need CloudFront config |
+| IAM integration | |
+
+#### Cost Estimate (10K apps, 100GB, 1TB egress)
+
+| Component | Monthly Cost |
+|-----------|--------------|
+| Storage (100GB) | $2.30 |
+| CloudFront (1TB) | $85 |
+| Requests | ~$5 |
+| **Total** | **~$92** |
+
+---
+
+### Option C: Backblaze B2 + Cloudflare
+
+```
+Storage:    Backblaze B2
+CDN:        Cloudflare (free egress via Bandwidth Alliance)
+Egress:     FREE (through Cloudflare)
+Pricing:    $0.005/GB storage
+```
+
+| Pros | Cons |
+|------|------|
+| Cheapest storage | Two services to manage |
+| Free egress via CF | B2 API slightly different |
+| Good reliability | Need CF proxy setup |
+
+#### Cost Estimate (10K apps, 100GB)
+
+| Component | Monthly Cost |
+|-----------|--------------|
+| Storage (100GB) | $0.50 |
+| Egress (via CF) | $0 |
+| Requests | ~$0.40 |
+| **Total** | **~$1** |
+
+---
+
+### Option D: Self-hosted MinIO
+
+```
+Storage:    MinIO on VPS
+CDN:        Cloudflare proxy
+Egress:     VPS bandwidth
+Pricing:    VPS cost only
+```
+
+| Pros | Cons |
+|------|------|
+| Full control | Ops overhead |
+| S3-compatible | Need to manage |
+| Predictable cost | Scaling complexity |
+
+#### Cost Estimate
+
+| Component | Monthly Cost |
+|-----------|--------------|
+| VPS (500GB SSD) | $20-50 |
+| Cloudflare | $0 (free tier) |
+| **Total** | **~$20-50** |
+
+---
+
+## Recommendation
+
+**Primary: Local Synology Filesystem**
+- Zero recurring costs
+- All data on premises
+- Simple Go file serving
+- Synology's built-in backup tools
+
+**Optional: Cloudflare R2 for CDN**
+- If global distribution needed
+- Sync packages via cron/background job
+- Zero egress fees
+- S3-compatible API
+
+---
+
+## Upload Flow
+
+### Local Filesystem Approach
+
+```
+┌────────┐    ┌─────────┐    ┌─────────────────┐
+│ Client │───►│   API   │───►│ Local Filesystem│
+└────────┘    └─────────┘    └─────────────────┘
+     │              │                  │
+     │  1. Upload package (multipart)  │
+     │─────────────►│                  │
+     │              │                  │
+     │              │  2. Save to temp │
+     │              │─────────────────►│
+     │              │                  │
+     │              │  3. Validate     │
+     │              │  4. Move to final│
+     │              │─────────────────►│
+     │              │                  │
+     │  5. Confirm  │                  │
+     │◄─────────────┤                  │
+```
+
+### Go Implementation (Local Storage)
+
+```go
+// Upload handler - direct file upload
+func (h *VersionHandler) Upload(w http.ResponseWriter, r *http.Request) {
+    appID := chi.URLParam(r, "appID")
+    devID := r.Context().Value("developer_id").(string)
+
+    // Parse multipart form (max 50MB)
+    r.ParseMultipartForm(50 << 20)
+    file, header, err := r.FormFile("package")
+    if err != nil {
+        respondError(w, http.StatusBadRequest, "INVALID_FILE", "No package file")
+        return
+    }
+    defer file.Close()
+
+    // Create version record
+    version := &domain.Version{
+        ID:          uuid.New().String(),
+        AppID:       appID,
+        VersionCode: parseInt(r.FormValue("version_code")),
+        VersionName: r.FormValue("version_name"),
+        Status:      domain.VersionStatusUploading,
+    }
+
+    // Save to temp directory
+    tempPath := filepath.Join(h.storagePath, "temp", version.ID, "package.mosis")
+    os.MkdirAll(filepath.Dir(tempPath), 0755)
+
+    dst, err := os.Create(tempPath)
+    if err != nil {
+        respondError(w, http.StatusInternalServerError, "STORAGE_ERROR", err.Error())
+        return
+    }
+    defer dst.Close()
+
+    size, err := io.Copy(dst, file)
+    version.PackageSize = size
+
+    // Validate package (signature, manifest, etc.)
+    if err := h.validator.Validate(tempPath); err != nil {
+        os.Remove(tempPath)
+        respondError(w, http.StatusBadRequest, "VALIDATION_ERROR", err.Error())
+        return
+    }
+
+    // Move to final location
+    finalPath := filepath.Join(h.storagePath, "packages",
+        devID, appID, fmt.Sprintf("%d", version.VersionCode), "package.mosis")
+    os.MkdirAll(filepath.Dir(finalPath), 0755)
+    os.Rename(tempPath, finalPath)
+
+    version.PackageURL = finalPath
+    version.Status = domain.VersionStatusDraft
+    h.repo.Save(version)
+
+    respondJSON(w, http.StatusCreated, version)
+}
+```
+
+### Optional: Presigned URL for R2 CDN
+
+If global distribution is needed, sync to R2 and generate presigned download URLs:
+
+```go
+// Sync to R2 after publishing (background job)
+func (s *SyncService) SyncToR2(version *domain.Version) error {
+    localPath := version.PackageURL
+    r2Key := fmt.Sprintf("packages/%s/%s/%d/package.mosis",
+        version.DeveloperID, version.AppID, version.VersionCode)
+
+    file, _ := os.Open(localPath)
+    defer file.Close()
+
+    _, err := s.r2.PutObject(r2Key, file)
+    return err
+}
+```
+
+---
+
+## Download Flow
+
+### Local File Serving
+
+```go
+// Serve package downloads directly from filesystem
+func (h *StoreHandler) Download(w http.ResponseWriter, r *http.Request) {
+    packageID := chi.URLParam(r, "packageID")
+
+    // Get latest published version
+    version, err := h.repo.GetLatestPublished(packageID)
+    if err != nil {
+        respondError(w, http.StatusNotFound, "NOT_FOUND", "App not found")
+        return
+    }
+
+    // Serve file directly
+    filePath := version.PackageURL
+    file, err := os.Open(filePath)
+    if err != nil {
+        respondError(w, http.StatusNotFound, "FILE_NOT_FOUND", "Package not available")
+        return
+    }
+    defer file.Close()
+
+    stat, _ := file.Stat()
+
+    // Set headers
+    w.Header().Set("Content-Type", "application/octet-stream")
+    w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%s-%s.mosis",
+        packageID, version.VersionName))
+    w.Header().Set("Content-Length", fmt.Sprintf("%d", stat.Size()))
+    w.Header().Set("X-Mosis-Version", version.VersionName)
+    w.Header().Set("X-Mosis-Signature", version.Signature)
+
+    // Stream file
+    io.Copy(w, file)
+}
+
+// Alternative: Return download info + serve via static file handler
+func (h *StoreHandler) GetDownloadInfo(w http.ResponseWriter, r *http.Request) {
+    packageID := chi.URLParam(r, "packageID")
+    version, _ := h.repo.GetLatestPublished(packageID)
+
+    respondJSON(w, http.StatusOK, map[string]interface{}{
+        "download_url": fmt.Sprintf("/downloads/%s/%s/%d/package.mosis",
+            version.DeveloperID, version.AppID, version.VersionCode),
+        "version":   version.VersionName,
+        "size":      version.PackageSize,
+        "signature": version.Signature,
+    })
+}
+```
+
+### Static File Server
+
+```go
+// In router setup - serve packages directory
+r.Handle("/downloads/*", http.StripPrefix("/downloads/",
+    http.FileServer(http.Dir("/volume1/mosis/packages"))))
+```
+
+### Caching (via Nginx or Cloudflare Tunnel)
+
+```
+Cache-Control: public, max-age=86400
+```
+
+- Packages are immutable (version code = unique)
+- Put Nginx in front for caching if needed
+- Or use Cloudflare Tunnel for edge caching
+
+---
+
+## Icon/Screenshot Handling
+
+### Upload (Go)
+
+```go
+// Icons uploaded with app creation/update
+func (h *AppHandler) UploadIcon(w http.ResponseWriter, r *http.Request) {
+    appID := chi.URLParam(r, "appID")
+
+    r.ParseMultipartForm(10 << 20) // 10MB max
+    file, _, err := r.FormFile("icon")
+    if err != nil {
+        respondError(w, http.StatusBadRequest, "INVALID_FILE", "No icon file")
+        return
+    }
+    defer file.Close()
+
+    // Decode and validate
+    img, _, err := image.Decode(file)
+    if err != nil {
+        respondError(w, http.StatusBadRequest, "INVALID_IMAGE", "Cannot decode image")
+        return
+    }
+
+    bounds := img.Bounds()
+    if bounds.Dx() != bounds.Dy() {
+        respondError(w, http.StatusBadRequest, "INVALID_DIMENSIONS", "Icon must be square")
+        return
+    }
+
+    // Generate multiple sizes
+    sizes := []int{32, 64, 128}
+    assetsDir := filepath.Join(h.storagePath, "assets", appID)
+    os.MkdirAll(assetsDir, 0755)
+
+    for _, size := range sizes {
+        resized := resize.Resize(uint(size), uint(size), img, resize.Lanczos3)
+        outPath := filepath.Join(assetsDir, fmt.Sprintf("icon-%d.png", size))
+
+        out, _ := os.Create(outPath)
+        png.Encode(out, resized)
+        out.Close()
+    }
+
+    respondJSON(w, http.StatusOK, map[string]string{"status": "uploaded"})
+}
+```
+
+### Serving
+
+```go
+// Serve assets via static file handler
+r.Handle("/assets/*", http.StripPrefix("/assets/",
+    http.FileServer(http.Dir("/volume1/mosis/assets"))))
+```
+
+URLs:
+```
+https://portal.mosis.local/assets/{app_id}/icon-64.png
+```
+
+- Public read access
+- Long cache TTL (icons rarely change)
+- Nginx caching if needed
+
+---
+
+## Retention Policy
+
+### Packages
+
+| Status | Retention |
+|--------|-----------|
+| Published (current) | Forever |
+| Published (old) | 1 year |
+| Draft | 30 days |
+| Rejected | 7 days |
+| Failed validation | 24 hours |
+
+### Temp Uploads
+
+- Delete after 1 hour if not completed
+- Lifecycle rule on `/temp/` prefix
+
+### Implementation
+
+```yaml
+# R2 Lifecycle Rules
+rules:
+  - prefix: "temp/"
+    expiration_days: 1
+
+  - prefix: "packages/"
+    tags:
+      status: draft
+    expiration_days: 30
+```
+
+---
+
+## Backup Strategy
+
+### Synology Built-in Options
+
+- **Hyper Backup** - Backup to external drive, another NAS, or cloud
+- **Snapshot Replication** - Point-in-time snapshots (Btrfs)
+- **rsync** - Script-based backup to remote location
+
+### Recommended Setup
+
+```bash
+# Cron job: Daily backup of packages to external drive
+0 3 * * * rsync -av /volume1/mosis/packages/ /volumeUSB1/mosis-backup/packages/
+
+# Or use Synology Hyper Backup with versioning
+```
+
+### Metadata Backup
+
+- Package metadata in SQLite (portal.db)
+- Litestream handles continuous replication
+- Can regenerate file paths from DB
+
+### Recovery
+
+```bash
+# Restore from backup
+rsync -av /volumeUSB1/mosis-backup/packages/ /volume1/mosis/packages/
+
+# Restore database (via Litestream)
+litestream restore -o /data/portal.db /backups/portal/
+```
+
+---
+
+## Monitoring
+
+### Metrics to Track
+
+| Metric | Source |
+|--------|--------|
+| Upload success rate | API logs |
+| Download latency | Cloudflare analytics |
+| Storage usage | R2 dashboard |
+| Bandwidth | R2 dashboard |
+| Error rate | API logs |
+
+### Alerts
+
+- Upload failure rate > 5%
+- Download error rate > 1%
+- Storage > 80% of budget
+
+---
+
+## Security
+
+### Access Control
+
+```
+Packages:
+  - Write: API only (presigned URLs)
+  - Read: Public (presigned URLs)
+
+Assets:
+  - Write: API only
+  - Read: Public (direct CDN)
+
+Temp:
+  - Write: Presigned URLs (15 min)
+  - Read: None (API internal only)
+```
+
+### Signed URLs
+
+```go
+// Presigned URL with expiration
+url := r2.PresignPut(key, 15*time.Minute, PutOptions{
+    ContentType:   "application/octet-stream",
+    ContentLength: maxSize,
+})
+```
+
+---
+
+## Deliverables
+
+- [x] Storage approach decided (local Synology filesystem)
+- [ ] Upload flow implementation (multipart to local)
+- [ ] Download serving (http.FileServer)
+- [ ] Icon/screenshot upload and resize
+- [ ] Temp file cleanup (cron job)
+- [ ] Backup setup (Hyper Backup or rsync)
+- [ ] (Optional) R2 sync for CDN
+
+---
+
+## Open Questions
+
+1. ~~Multi-region storage for lower latency?~~ → Use R2 sync if needed
+2. ~~Package compression (gzip/brotli)?~~ → Defer, .mosis is already ZIP
+3. Delta updates storage structure? → Consider for v1.1
+4. ~~Screenshot requirements (dimensions, count)?~~ → Max 5, 1280x720 or 720x1280
+
+---
+
+## References
+
+- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/)
+- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
+- [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html)