628 lines
17 KiB
Markdown
628 lines
17 KiB
Markdown
# Milestone 7: CDN & Storage
|
|
|
|
**Status**: Decided
|
|
**Goal**: Scalable storage for app packages and assets with global distribution.
|
|
|
|
## Decision
|
|
|
|
**Local Synology filesystem** as primary storage, with optional Cloudflare R2 for CDN:
|
|
|
|
```
|
|
Primary: Synology volume (/volume1/mosis/)
|
|
CDN: Cloudflare R2 (optional, for global distribution)
|
|
Serving: Go binary serves files directly (local)
|
|
Backup: Synology Hyper Backup / rsync
|
|
```
|
|
|
|
### Rationale
|
|
|
|
1. **Self-hosted** - All data stays on premises
|
|
2. **Zero cost** - No cloud storage fees
|
|
3. **Simple** - Go binary serves files directly via http.FileServer
|
|
4. **Fast local** - NAS has gigabit+ internal network
|
|
5. **Optional CDN** - Can sync to R2 if global distribution needed
|
|
|
|
### Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Synology NAS │
|
|
│ ┌────────────────────────────────────────────────────────────┐ │
|
|
│ │ mosis-portal container │ │
|
|
│ │ │ │
|
|
│ │ Go binary ──serves──► /packages/, /assets/ │ │
|
|
│ │ │ │ │
|
|
│ │ └── SQLite (/data/portal.db) │ │
|
|
│ └──────────────────────────┬─────────────────────────────────┘ │
|
|
│ │ │
|
|
│ /volume1/mosis/ │ bind mount │
|
|
│ ├── data/portal.db │ │
|
|
│ ├── packages/ ◄───────────┘ │
|
|
│ │ └── {dev_id}/{app_id}/{version}/package.mosis │
|
|
│ ├── assets/ │
|
|
│ │ └── {app_id}/icon-{size}.png │
|
|
│ └── backups/ │
|
|
│ └── litestream replicas │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
(optional sync)
|
|
▼
|
|
┌──────────────────┐
|
|
│ Cloudflare R2 │
|
|
│ (CDN for global │
|
|
│ distribution) │
|
|
└──────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Storage handles app packages, icons, screenshots, and serves downloads to devices. For self-hosted Synology NAS deployment, local filesystem is the primary storage.
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
### Functional
|
|
|
|
- Store app packages (.mosis files)
|
|
- Store icons (multiple sizes)
|
|
- Store screenshots (optional)
|
|
- Serve downloads globally
|
|
- Support presigned upload URLs
|
|
- Version retention (keep old versions)
|
|
|
|
### Non-Functional
|
|
|
|
| Requirement | Target |
|
|
|-------------|--------|
|
|
| Upload speed | < 30s for 50MB |
|
|
| Download latency | < 100ms (p50) |
|
|
| Availability | 99.9% |
|
|
| Durability | 99.999999999% (11 nines) |
|
|
| Cost | Minimize egress fees |
|
|
|
|
---
|
|
|
|
## Storage Structure
|
|
|
|
```
|
|
/packages/
|
|
/{developer_id}/
|
|
/{app_id}/
|
|
/{version_code}/
|
|
package.mosis
|
|
manifest.json (extracted for quick access)
|
|
|
|
/assets/
|
|
/{app_id}/
|
|
icon-32.png
|
|
icon-64.png
|
|
icon-128.png
|
|
screenshots/
|
|
1.png
|
|
2.png
|
|
3.png
|
|
|
|
/temp/
|
|
/{upload_id}/
|
|
package.mosis (pending validation)
|
|
```
|
|
|
|
---
|
|
|
|
## Options Analysis
|
|
|
|
### Option A: Cloudflare R2
|
|
|
|
```
|
|
Storage: Object storage (S3-compatible)
|
|
CDN: Cloudflare network (automatic)
|
|
Egress: FREE (zero egress fees)
|
|
Pricing: $0.015/GB storage, $4.50/million requests
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| No egress fees | Newer service |
|
|
| Global CDN included | Fewer regions than S3 |
|
|
| S3-compatible API | Less tooling |
|
|
| Workers integration | |
|
|
|
|
#### Cost Estimate (10K apps, 100GB)
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| Storage (100GB) | $1.50 |
|
|
| Requests (1M) | $4.50 |
|
|
| Egress | $0 |
|
|
| **Total** | **~$6** |
|
|
|
|
---
|
|
|
|
### Option B: AWS S3 + CloudFront
|
|
|
|
```
|
|
Storage: S3 Standard
|
|
CDN: CloudFront
|
|
Egress: $0.085-0.12/GB (varies by region)
|
|
Pricing: $0.023/GB storage
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| Most mature | Egress costs add up |
|
|
| Best tooling | Complex pricing |
|
|
| All regions | Need CloudFront config |
|
|
| IAM integration | |
|
|
|
|
#### Cost Estimate (10K apps, 100GB, 1TB egress)
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| Storage (100GB) | $2.30 |
|
|
| CloudFront (1TB) | $85 |
|
|
| Requests | ~$5 |
|
|
| **Total** | **~$92** |
|
|
|
|
---
|
|
|
|
### Option C: Backblaze B2 + Cloudflare
|
|
|
|
```
|
|
Storage: Backblaze B2
|
|
CDN: Cloudflare (free egress via Bandwidth Alliance)
|
|
Egress: FREE (through Cloudflare)
|
|
Pricing: $0.005/GB storage
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| Cheapest storage | Two services to manage |
|
|
| Free egress via CF | B2 API slightly different |
|
|
| Good reliability | Need CF proxy setup |
|
|
|
|
#### Cost Estimate (10K apps, 100GB)
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| Storage (100GB) | $0.50 |
|
|
| Egress (via CF) | $0 |
|
|
| Requests | ~$0.40 |
|
|
| **Total** | **~$1** |
|
|
|
|
---
|
|
|
|
### Option D: Self-hosted MinIO
|
|
|
|
```
|
|
Storage: MinIO on VPS
|
|
CDN: Cloudflare proxy
|
|
Egress: VPS bandwidth
|
|
Pricing: VPS cost only
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| Full control | Ops overhead |
|
|
| S3-compatible | Need to manage |
|
|
| Predictable cost | Scaling complexity |
|
|
|
|
#### Cost Estimate
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| VPS (500GB SSD) | $20-50 |
|
|
| Cloudflare | $0 (free tier) |
|
|
| **Total** | **~$20-50** |
|
|
|
|
---
|
|
|
|
## Recommendation
|
|
|
|
**Primary: Local Synology Filesystem**
|
|
- Zero recurring costs
|
|
- All data on premises
|
|
- Simple Go file serving
|
|
- Synology's built-in backup tools
|
|
|
|
**Optional: Cloudflare R2 for CDN**
|
|
- If global distribution needed
|
|
- Sync packages via cron/background job
|
|
- Zero egress fees
|
|
- S3-compatible API
|
|
|
|
---
|
|
|
|
## Upload Flow
|
|
|
|
### Local Filesystem Approach
|
|
|
|
```
|
|
┌────────┐ ┌─────────┐ ┌─────────────────┐
|
|
│ Client │───►│ API │───►│ Local Filesystem│
|
|
└────────┘ └─────────┘ └─────────────────┘
|
|
│ │ │
|
|
│ 1. Upload package (multipart) │
|
|
│─────────────►│ │
|
|
│ │ │
|
|
│ │ 2. Save to temp │
|
|
│ │─────────────────►│
|
|
│ │ │
|
|
│ │ 3. Validate │
|
|
│ │ 4. Move to final│
|
|
│ │─────────────────►│
|
|
│ │ │
|
|
│ 5. Confirm │ │
|
|
│◄─────────────┤ │
|
|
```
|
|
|
|
### Go Implementation (Local Storage)
|
|
|
|
```go
|
|
// Upload handler - direct file upload
|
|
func (h *VersionHandler) Upload(w http.ResponseWriter, r *http.Request) {
|
|
appID := chi.URLParam(r, "appID")
|
|
devID := r.Context().Value("developer_id").(string)
|
|
|
|
// Parse multipart form (max 50MB)
|
|
r.ParseMultipartForm(50 << 20)
|
|
file, header, err := r.FormFile("package")
|
|
if err != nil {
|
|
respondError(w, http.StatusBadRequest, "INVALID_FILE", "No package file")
|
|
return
|
|
}
|
|
defer file.Close()
|
|
|
|
// Create version record
|
|
version := &domain.Version{
|
|
ID: uuid.New().String(),
|
|
AppID: appID,
|
|
VersionCode: parseInt(r.FormValue("version_code")),
|
|
VersionName: r.FormValue("version_name"),
|
|
Status: domain.VersionStatusUploading,
|
|
}
|
|
|
|
// Save to temp directory
|
|
tempPath := filepath.Join(h.storagePath, "temp", version.ID, "package.mosis")
|
|
os.MkdirAll(filepath.Dir(tempPath), 0755)
|
|
|
|
dst, err := os.Create(tempPath)
|
|
if err != nil {
|
|
respondError(w, http.StatusInternalServerError, "STORAGE_ERROR", err.Error())
|
|
return
|
|
}
|
|
defer dst.Close()
|
|
|
|
size, err := io.Copy(dst, file)
|
|
version.PackageSize = size
|
|
|
|
// Validate package (signature, manifest, etc.)
|
|
if err := h.validator.Validate(tempPath); err != nil {
|
|
os.Remove(tempPath)
|
|
respondError(w, http.StatusBadRequest, "VALIDATION_ERROR", err.Error())
|
|
return
|
|
}
|
|
|
|
// Move to final location
|
|
finalPath := filepath.Join(h.storagePath, "packages",
|
|
devID, appID, fmt.Sprintf("%d", version.VersionCode), "package.mosis")
|
|
os.MkdirAll(filepath.Dir(finalPath), 0755)
|
|
os.Rename(tempPath, finalPath)
|
|
|
|
version.PackageURL = finalPath
|
|
version.Status = domain.VersionStatusDraft
|
|
h.repo.Save(version)
|
|
|
|
respondJSON(w, http.StatusCreated, version)
|
|
}
|
|
```
|
|
|
|
### Optional: Presigned URL for R2 CDN
|
|
|
|
If global distribution is needed, sync to R2 and generate presigned download URLs:
|
|
|
|
```go
|
|
// Sync to R2 after publishing (background job)
|
|
func (s *SyncService) SyncToR2(version *domain.Version) error {
|
|
localPath := version.PackageURL
|
|
r2Key := fmt.Sprintf("packages/%s/%s/%d/package.mosis",
|
|
version.DeveloperID, version.AppID, version.VersionCode)
|
|
|
|
file, _ := os.Open(localPath)
|
|
defer file.Close()
|
|
|
|
_, err := s.r2.PutObject(r2Key, file)
|
|
return err
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Download Flow
|
|
|
|
### Local File Serving
|
|
|
|
```go
|
|
// Serve package downloads directly from filesystem
|
|
func (h *StoreHandler) Download(w http.ResponseWriter, r *http.Request) {
|
|
packageID := chi.URLParam(r, "packageID")
|
|
|
|
// Get latest published version
|
|
version, err := h.repo.GetLatestPublished(packageID)
|
|
if err != nil {
|
|
respondError(w, http.StatusNotFound, "NOT_FOUND", "App not found")
|
|
return
|
|
}
|
|
|
|
// Serve file directly
|
|
filePath := version.PackageURL
|
|
file, err := os.Open(filePath)
|
|
if err != nil {
|
|
respondError(w, http.StatusNotFound, "FILE_NOT_FOUND", "Package not available")
|
|
return
|
|
}
|
|
defer file.Close()
|
|
|
|
stat, _ := file.Stat()
|
|
|
|
// Set headers
|
|
w.Header().Set("Content-Type", "application/octet-stream")
|
|
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%s-%s.mosis",
|
|
packageID, version.VersionName))
|
|
w.Header().Set("Content-Length", fmt.Sprintf("%d", stat.Size()))
|
|
w.Header().Set("X-Mosis-Version", version.VersionName)
|
|
w.Header().Set("X-Mosis-Signature", version.Signature)
|
|
|
|
// Stream file
|
|
io.Copy(w, file)
|
|
}
|
|
|
|
// Alternative: Return download info + serve via static file handler
|
|
func (h *StoreHandler) GetDownloadInfo(w http.ResponseWriter, r *http.Request) {
|
|
packageID := chi.URLParam(r, "packageID")
|
|
version, _ := h.repo.GetLatestPublished(packageID)
|
|
|
|
respondJSON(w, http.StatusOK, map[string]interface{}{
|
|
"download_url": fmt.Sprintf("/downloads/%s/%s/%d/package.mosis",
|
|
version.DeveloperID, version.AppID, version.VersionCode),
|
|
"version": version.VersionName,
|
|
"size": version.PackageSize,
|
|
"signature": version.Signature,
|
|
})
|
|
}
|
|
```
|
|
|
|
### Static File Server
|
|
|
|
```go
|
|
// In router setup - serve packages directory
|
|
r.Handle("/downloads/*", http.StripPrefix("/downloads/",
|
|
http.FileServer(http.Dir("/volume1/mosis/packages"))))
|
|
```
|
|
|
|
### Caching (via Nginx or Cloudflare Tunnel)
|
|
|
|
```
|
|
Cache-Control: public, max-age=86400
|
|
```
|
|
|
|
- Packages are immutable (version code = unique)
|
|
- Put Nginx in front for caching if needed
|
|
- Or use Cloudflare Tunnel for edge caching
|
|
|
|
---
|
|
|
|
## Icon/Screenshot Handling
|
|
|
|
### Upload (Go)
|
|
|
|
```go
|
|
// Icons uploaded with app creation/update
|
|
func (h *AppHandler) UploadIcon(w http.ResponseWriter, r *http.Request) {
|
|
appID := chi.URLParam(r, "appID")
|
|
|
|
r.ParseMultipartForm(10 << 20) // 10MB max
|
|
file, _, err := r.FormFile("icon")
|
|
if err != nil {
|
|
respondError(w, http.StatusBadRequest, "INVALID_FILE", "No icon file")
|
|
return
|
|
}
|
|
defer file.Close()
|
|
|
|
// Decode and validate
|
|
img, _, err := image.Decode(file)
|
|
if err != nil {
|
|
respondError(w, http.StatusBadRequest, "INVALID_IMAGE", "Cannot decode image")
|
|
return
|
|
}
|
|
|
|
bounds := img.Bounds()
|
|
if bounds.Dx() != bounds.Dy() {
|
|
respondError(w, http.StatusBadRequest, "INVALID_DIMENSIONS", "Icon must be square")
|
|
return
|
|
}
|
|
|
|
// Generate multiple sizes
|
|
sizes := []int{32, 64, 128}
|
|
assetsDir := filepath.Join(h.storagePath, "assets", appID)
|
|
os.MkdirAll(assetsDir, 0755)
|
|
|
|
for _, size := range sizes {
|
|
resized := resize.Resize(uint(size), uint(size), img, resize.Lanczos3)
|
|
outPath := filepath.Join(assetsDir, fmt.Sprintf("icon-%d.png", size))
|
|
|
|
out, _ := os.Create(outPath)
|
|
png.Encode(out, resized)
|
|
out.Close()
|
|
}
|
|
|
|
respondJSON(w, http.StatusOK, map[string]string{"status": "uploaded"})
|
|
}
|
|
```
|
|
|
|
### Serving
|
|
|
|
```go
|
|
// Serve assets via static file handler
|
|
r.Handle("/assets/*", http.StripPrefix("/assets/",
|
|
http.FileServer(http.Dir("/volume1/mosis/assets"))))
|
|
```
|
|
|
|
URLs:
|
|
```
|
|
https://portal.mosis.local/assets/{app_id}/icon-64.png
|
|
```
|
|
|
|
- Public read access
|
|
- Long cache TTL (icons rarely change)
|
|
- Nginx caching if needed
|
|
|
|
---
|
|
|
|
## Retention Policy
|
|
|
|
### Packages
|
|
|
|
| Status | Retention |
|
|
|--------|-----------|
|
|
| Published (current) | Forever |
|
|
| Published (old) | 1 year |
|
|
| Draft | 30 days |
|
|
| Rejected | 7 days |
|
|
| Failed validation | 24 hours |
|
|
|
|
### Temp Uploads
|
|
|
|
- Delete after 1 hour if not completed
|
|
- Lifecycle rule on `/temp/` prefix
|
|
|
|
### Implementation
|
|
|
|
```yaml
|
|
# R2 Lifecycle Rules
|
|
rules:
|
|
- prefix: "temp/"
|
|
expiration_days: 1
|
|
|
|
- prefix: "packages/"
|
|
tags:
|
|
status: draft
|
|
expiration_days: 30
|
|
```
|
|
|
|
---
|
|
|
|
## Backup Strategy
|
|
|
|
### Synology Built-in Options
|
|
|
|
- **Hyper Backup** - Backup to external drive, another NAS, or cloud
|
|
- **Snapshot Replication** - Point-in-time snapshots (Btrfs)
|
|
- **rsync** - Script-based backup to remote location
|
|
|
|
### Recommended Setup
|
|
|
|
```bash
|
|
# Cron job: Daily backup of packages to external drive
|
|
0 3 * * * rsync -av /volume1/mosis/packages/ /volumeUSB1/mosis-backup/packages/
|
|
|
|
# Or use Synology Hyper Backup with versioning
|
|
```
|
|
|
|
### Metadata Backup
|
|
|
|
- Package metadata in SQLite (portal.db)
|
|
- Litestream handles continuous replication
|
|
- Can regenerate file paths from DB
|
|
|
|
### Recovery
|
|
|
|
```bash
|
|
# Restore from backup
|
|
rsync -av /volumeUSB1/mosis-backup/packages/ /volume1/mosis/packages/
|
|
|
|
# Restore database (via Litestream)
|
|
litestream restore -o /data/portal.db /backups/portal/
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring
|
|
|
|
### Metrics to Track
|
|
|
|
| Metric | Source |
|
|
|--------|--------|
|
|
| Upload success rate | API logs |
|
|
| Download latency | Cloudflare analytics |
|
|
| Storage usage | R2 dashboard |
|
|
| Bandwidth | R2 dashboard |
|
|
| Error rate | API logs |
|
|
|
|
### Alerts
|
|
|
|
- Upload failure rate > 5%
|
|
- Download error rate > 1%
|
|
- Storage > 80% of budget
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
### Access Control
|
|
|
|
```
|
|
Packages:
|
|
- Write: API only (presigned URLs)
|
|
- Read: Public (presigned URLs)
|
|
|
|
Assets:
|
|
- Write: API only
|
|
- Read: Public (direct CDN)
|
|
|
|
Temp:
|
|
- Write: Presigned URLs (15 min)
|
|
- Read: None (API internal only)
|
|
```
|
|
|
|
### Signed URLs
|
|
|
|
```go
|
|
// Presigned URL with expiration
|
|
url := r2.PresignPut(key, 15*time.Minute, PutOptions{
|
|
ContentType: "application/octet-stream",
|
|
ContentLength: maxSize,
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
## Deliverables
|
|
|
|
- [x] Storage approach decided (local Synology filesystem)
|
|
- [ ] Upload flow implementation (multipart to local)
|
|
- [ ] Download serving (http.FileServer)
|
|
- [ ] Icon/screenshot upload and resize
|
|
- [ ] Temp file cleanup (cron job)
|
|
- [ ] Backup setup (Hyper Backup or rsync)
|
|
- [ ] (Optional) R2 sync for CDN
|
|
|
|
---
|
|
|
|
## Open Questions
|
|
|
|
1. ~~Multi-region storage for lower latency?~~ → Use R2 sync if needed
|
|
2. ~~Package compression (gzip/brotli)?~~ → Defer, .mosis is already ZIP
|
|
3. Delta updates storage structure? → Consider for v1.1
|
|
4. ~~Screenshot requirements (dimensions, count)?~~ → Max 5, 1280x720 or 720x1280
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/)
|
|
- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
|
|
- [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html)
|