# Milestone 7: CDN & Storage **Status**: Decided **Goal**: Scalable storage for app packages and assets with global distribution. ## Decision **Local Synology filesystem** as primary storage, with optional Cloudflare R2 for CDN: ``` Primary: Synology volume (/volume1/mosis/) CDN: Cloudflare R2 (optional, for global distribution) Serving: Go binary serves files directly (local) Backup: Synology Hyper Backup / rsync ``` ### Rationale 1. **Self-hosted** - All data stays on premises 2. **Zero cost** - No cloud storage fees 3. **Simple** - Go binary serves files directly via http.FileServer 4. **Fast local** - NAS has gigabit+ internal network 5. **Optional CDN** - Can sync to R2 if global distribution needed ### Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Synology NAS │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ mosis-portal container │ │ │ │ │ │ │ │ Go binary ──serves──► /packages/, /assets/ │ │ │ │ │ │ │ │ │ └── SQLite (/data/portal.db) │ │ │ └──────────────────────────┬─────────────────────────────────┘ │ │ │ │ │ /volume1/mosis/ │ bind mount │ │ ├── data/portal.db │ │ │ ├── packages/ ◄───────────┘ │ │ │ └── {dev_id}/{app_id}/{version}/package.mosis │ │ ├── assets/ │ │ │ └── {app_id}/icon-{size}.png │ │ └── backups/ │ │ └── litestream replicas │ └─────────────────────────────────────────────────────────────────┘ │ (optional sync) ▼ ┌──────────────────┐ │ Cloudflare R2 │ │ (CDN for global │ │ distribution) │ └──────────────────┘ ``` --- ## Overview Storage handles app packages, icons, screenshots, and serves downloads to devices. For self-hosted Synology NAS deployment, local filesystem is the primary storage. --- ## Requirements ### Functional - Store app packages (.mosis files) - Store icons (multiple sizes) - Store screenshots (optional) - Serve downloads globally - Support presigned upload URLs - Version retention (keep old versions) ### Non-Functional | Requirement | Target | |-------------|--------| | Upload speed | < 30s for 50MB | | Download latency | < 100ms (p50) | | Availability | 99.9% | | Durability | 99.999999999% (11 nines) | | Cost | Minimize egress fees | --- ## Storage Structure ``` /packages/ /{developer_id}/ /{app_id}/ /{version_code}/ package.mosis manifest.json (extracted for quick access) /assets/ /{app_id}/ icon-32.png icon-64.png icon-128.png screenshots/ 1.png 2.png 3.png /temp/ /{upload_id}/ package.mosis (pending validation) ``` --- ## Options Analysis ### Option A: Cloudflare R2 ``` Storage: Object storage (S3-compatible) CDN: Cloudflare network (automatic) Egress: FREE (zero egress fees) Pricing: $0.015/GB storage, $4.50/million requests ``` | Pros | Cons | |------|------| | No egress fees | Newer service | | Global CDN included | Fewer regions than S3 | | S3-compatible API | Less tooling | | Workers integration | | #### Cost Estimate (10K apps, 100GB) | Component | Monthly Cost | |-----------|--------------| | Storage (100GB) | $1.50 | | Requests (1M) | $4.50 | | Egress | $0 | | **Total** | **~$6** | --- ### Option B: AWS S3 + CloudFront ``` Storage: S3 Standard CDN: CloudFront Egress: $0.085-0.12/GB (varies by region) Pricing: $0.023/GB storage ``` | Pros | Cons | |------|------| | Most mature | Egress costs add up | | Best tooling | Complex pricing | | All regions | Need CloudFront config | | IAM integration | | #### Cost Estimate (10K apps, 100GB, 1TB egress) | Component | Monthly Cost | |-----------|--------------| | Storage (100GB) | $2.30 | | CloudFront (1TB) | $85 | | Requests | ~$5 | | **Total** | **~$92** | --- ### Option C: Backblaze B2 + Cloudflare ``` Storage: Backblaze B2 CDN: Cloudflare (free egress via Bandwidth Alliance) Egress: FREE (through Cloudflare) Pricing: $0.005/GB storage ``` | Pros | Cons | |------|------| | Cheapest storage | Two services to manage | | Free egress via CF | B2 API slightly different | | Good reliability | Need CF proxy setup | #### Cost Estimate (10K apps, 100GB) | Component | Monthly Cost | |-----------|--------------| | Storage (100GB) | $0.50 | | Egress (via CF) | $0 | | Requests | ~$0.40 | | **Total** | **~$1** | --- ### Option D: Self-hosted MinIO ``` Storage: MinIO on VPS CDN: Cloudflare proxy Egress: VPS bandwidth Pricing: VPS cost only ``` | Pros | Cons | |------|------| | Full control | Ops overhead | | S3-compatible | Need to manage | | Predictable cost | Scaling complexity | #### Cost Estimate | Component | Monthly Cost | |-----------|--------------| | VPS (500GB SSD) | $20-50 | | Cloudflare | $0 (free tier) | | **Total** | **~$20-50** | --- ## Recommendation **Primary: Local Synology Filesystem** - Zero recurring costs - All data on premises - Simple Go file serving - Synology's built-in backup tools **Optional: Cloudflare R2 for CDN** - If global distribution needed - Sync packages via cron/background job - Zero egress fees - S3-compatible API --- ## Upload Flow ### Local Filesystem Approach ``` ┌────────┐ ┌─────────┐ ┌─────────────────┐ │ Client │───►│ API │───►│ Local Filesystem│ └────────┘ └─────────┘ └─────────────────┘ │ │ │ │ 1. Upload package (multipart) │ │─────────────►│ │ │ │ │ │ │ 2. Save to temp │ │ │─────────────────►│ │ │ │ │ │ 3. Validate │ │ │ 4. Move to final│ │ │─────────────────►│ │ │ │ │ 5. Confirm │ │ │◄─────────────┤ │ ``` ### Go Implementation (Local Storage) ```go // Upload handler - direct file upload func (h *VersionHandler) Upload(w http.ResponseWriter, r *http.Request) { appID := chi.URLParam(r, "appID") devID := r.Context().Value("developer_id").(string) // Parse multipart form (max 50MB) r.ParseMultipartForm(50 << 20) file, header, err := r.FormFile("package") if err != nil { respondError(w, http.StatusBadRequest, "INVALID_FILE", "No package file") return } defer file.Close() // Create version record version := &domain.Version{ ID: uuid.New().String(), AppID: appID, VersionCode: parseInt(r.FormValue("version_code")), VersionName: r.FormValue("version_name"), Status: domain.VersionStatusUploading, } // Save to temp directory tempPath := filepath.Join(h.storagePath, "temp", version.ID, "package.mosis") os.MkdirAll(filepath.Dir(tempPath), 0755) dst, err := os.Create(tempPath) if err != nil { respondError(w, http.StatusInternalServerError, "STORAGE_ERROR", err.Error()) return } defer dst.Close() size, err := io.Copy(dst, file) version.PackageSize = size // Validate package (signature, manifest, etc.) if err := h.validator.Validate(tempPath); err != nil { os.Remove(tempPath) respondError(w, http.StatusBadRequest, "VALIDATION_ERROR", err.Error()) return } // Move to final location finalPath := filepath.Join(h.storagePath, "packages", devID, appID, fmt.Sprintf("%d", version.VersionCode), "package.mosis") os.MkdirAll(filepath.Dir(finalPath), 0755) os.Rename(tempPath, finalPath) version.PackageURL = finalPath version.Status = domain.VersionStatusDraft h.repo.Save(version) respondJSON(w, http.StatusCreated, version) } ``` ### Optional: Presigned URL for R2 CDN If global distribution is needed, sync to R2 and generate presigned download URLs: ```go // Sync to R2 after publishing (background job) func (s *SyncService) SyncToR2(version *domain.Version) error { localPath := version.PackageURL r2Key := fmt.Sprintf("packages/%s/%s/%d/package.mosis", version.DeveloperID, version.AppID, version.VersionCode) file, _ := os.Open(localPath) defer file.Close() _, err := s.r2.PutObject(r2Key, file) return err } ``` --- ## Download Flow ### Local File Serving ```go // Serve package downloads directly from filesystem func (h *StoreHandler) Download(w http.ResponseWriter, r *http.Request) { packageID := chi.URLParam(r, "packageID") // Get latest published version version, err := h.repo.GetLatestPublished(packageID) if err != nil { respondError(w, http.StatusNotFound, "NOT_FOUND", "App not found") return } // Serve file directly filePath := version.PackageURL file, err := os.Open(filePath) if err != nil { respondError(w, http.StatusNotFound, "FILE_NOT_FOUND", "Package not available") return } defer file.Close() stat, _ := file.Stat() // Set headers w.Header().Set("Content-Type", "application/octet-stream") w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%s-%s.mosis", packageID, version.VersionName)) w.Header().Set("Content-Length", fmt.Sprintf("%d", stat.Size())) w.Header().Set("X-Mosis-Version", version.VersionName) w.Header().Set("X-Mosis-Signature", version.Signature) // Stream file io.Copy(w, file) } // Alternative: Return download info + serve via static file handler func (h *StoreHandler) GetDownloadInfo(w http.ResponseWriter, r *http.Request) { packageID := chi.URLParam(r, "packageID") version, _ := h.repo.GetLatestPublished(packageID) respondJSON(w, http.StatusOK, map[string]interface{}{ "download_url": fmt.Sprintf("/downloads/%s/%s/%d/package.mosis", version.DeveloperID, version.AppID, version.VersionCode), "version": version.VersionName, "size": version.PackageSize, "signature": version.Signature, }) } ``` ### Static File Server ```go // In router setup - serve packages directory r.Handle("/downloads/*", http.StripPrefix("/downloads/", http.FileServer(http.Dir("/volume1/mosis/packages")))) ``` ### Caching (via Nginx or Cloudflare Tunnel) ``` Cache-Control: public, max-age=86400 ``` - Packages are immutable (version code = unique) - Put Nginx in front for caching if needed - Or use Cloudflare Tunnel for edge caching --- ## Icon/Screenshot Handling ### Upload (Go) ```go // Icons uploaded with app creation/update func (h *AppHandler) UploadIcon(w http.ResponseWriter, r *http.Request) { appID := chi.URLParam(r, "appID") r.ParseMultipartForm(10 << 20) // 10MB max file, _, err := r.FormFile("icon") if err != nil { respondError(w, http.StatusBadRequest, "INVALID_FILE", "No icon file") return } defer file.Close() // Decode and validate img, _, err := image.Decode(file) if err != nil { respondError(w, http.StatusBadRequest, "INVALID_IMAGE", "Cannot decode image") return } bounds := img.Bounds() if bounds.Dx() != bounds.Dy() { respondError(w, http.StatusBadRequest, "INVALID_DIMENSIONS", "Icon must be square") return } // Generate multiple sizes sizes := []int{32, 64, 128} assetsDir := filepath.Join(h.storagePath, "assets", appID) os.MkdirAll(assetsDir, 0755) for _, size := range sizes { resized := resize.Resize(uint(size), uint(size), img, resize.Lanczos3) outPath := filepath.Join(assetsDir, fmt.Sprintf("icon-%d.png", size)) out, _ := os.Create(outPath) png.Encode(out, resized) out.Close() } respondJSON(w, http.StatusOK, map[string]string{"status": "uploaded"}) } ``` ### Serving ```go // Serve assets via static file handler r.Handle("/assets/*", http.StripPrefix("/assets/", http.FileServer(http.Dir("/volume1/mosis/assets")))) ``` URLs: ``` https://portal.mosis.local/assets/{app_id}/icon-64.png ``` - Public read access - Long cache TTL (icons rarely change) - Nginx caching if needed --- ## Retention Policy ### Packages | Status | Retention | |--------|-----------| | Published (current) | Forever | | Published (old) | 1 year | | Draft | 30 days | | Rejected | 7 days | | Failed validation | 24 hours | ### Temp Uploads - Delete after 1 hour if not completed - Lifecycle rule on `/temp/` prefix ### Implementation ```yaml # R2 Lifecycle Rules rules: - prefix: "temp/" expiration_days: 1 - prefix: "packages/" tags: status: draft expiration_days: 30 ``` --- ## Backup Strategy ### Synology Built-in Options - **Hyper Backup** - Backup to external drive, another NAS, or cloud - **Snapshot Replication** - Point-in-time snapshots (Btrfs) - **rsync** - Script-based backup to remote location ### Recommended Setup ```bash # Cron job: Daily backup of packages to external drive 0 3 * * * rsync -av /volume1/mosis/packages/ /volumeUSB1/mosis-backup/packages/ # Or use Synology Hyper Backup with versioning ``` ### Metadata Backup - Package metadata in SQLite (portal.db) - Litestream handles continuous replication - Can regenerate file paths from DB ### Recovery ```bash # Restore from backup rsync -av /volumeUSB1/mosis-backup/packages/ /volume1/mosis/packages/ # Restore database (via Litestream) litestream restore -o /data/portal.db /backups/portal/ ``` --- ## Monitoring ### Metrics to Track | Metric | Source | |--------|--------| | Upload success rate | API logs | | Download latency | Cloudflare analytics | | Storage usage | R2 dashboard | | Bandwidth | R2 dashboard | | Error rate | API logs | ### Alerts - Upload failure rate > 5% - Download error rate > 1% - Storage > 80% of budget --- ## Security ### Access Control ``` Packages: - Write: API only (presigned URLs) - Read: Public (presigned URLs) Assets: - Write: API only - Read: Public (direct CDN) Temp: - Write: Presigned URLs (15 min) - Read: None (API internal only) ``` ### Signed URLs ```go // Presigned URL with expiration url := r2.PresignPut(key, 15*time.Minute, PutOptions{ ContentType: "application/octet-stream", ContentLength: maxSize, }) ``` --- ## Deliverables - [x] Storage approach decided (local Synology filesystem) - [ ] Upload flow implementation (multipart to local) - [ ] Download serving (http.FileServer) - [ ] Icon/screenshot upload and resize - [ ] Temp file cleanup (cron job) - [ ] Backup setup (Hyper Backup or rsync) - [ ] (Optional) R2 sync for CDN --- ## Open Questions 1. ~~Multi-region storage for lower latency?~~ → Use R2 sync if needed 2. ~~Package compression (gzip/brotli)?~~ → Defer, .mosis is already ZIP 3. Delta updates storage structure? → Consider for v1.1 4. ~~Screenshot requirements (dimensions, count)?~~ → Max 5, 1280x720 or 720x1280 --- ## References - [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/) - [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html) - [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html)