452 lines
9.3 KiB
Markdown
452 lines
9.3 KiB
Markdown
# Milestone 7: CDN & Storage
|
|
|
|
**Status**: Planning
|
|
**Goal**: Scalable storage for app packages and assets with global distribution.
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Storage handles app packages, icons, screenshots, and serves downloads to devices worldwide. Must be cost-effective, fast, and reliable.
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
### Functional
|
|
|
|
- Store app packages (.mosis files)
|
|
- Store icons (multiple sizes)
|
|
- Store screenshots (optional)
|
|
- Serve downloads globally
|
|
- Support presigned upload URLs
|
|
- Version retention (keep old versions)
|
|
|
|
### Non-Functional
|
|
|
|
| Requirement | Target |
|
|
|-------------|--------|
|
|
| Upload speed | < 30s for 50MB |
|
|
| Download latency | < 100ms (p50) |
|
|
| Availability | 99.9% |
|
|
| Durability | 99.999999999% (11 nines) |
|
|
| Cost | Minimize egress fees |
|
|
|
|
---
|
|
|
|
## Storage Structure
|
|
|
|
```
|
|
/packages/
|
|
/{developer_id}/
|
|
/{app_id}/
|
|
/{version_code}/
|
|
package.mosis
|
|
manifest.json (extracted for quick access)
|
|
|
|
/assets/
|
|
/{app_id}/
|
|
icon-32.png
|
|
icon-64.png
|
|
icon-128.png
|
|
screenshots/
|
|
1.png
|
|
2.png
|
|
3.png
|
|
|
|
/temp/
|
|
/{upload_id}/
|
|
package.mosis (pending validation)
|
|
```
|
|
|
|
---
|
|
|
|
## Options Analysis
|
|
|
|
### Option A: Cloudflare R2
|
|
|
|
```
|
|
Storage: Object storage (S3-compatible)
|
|
CDN: Cloudflare network (automatic)
|
|
Egress: FREE (zero egress fees)
|
|
Pricing: $0.015/GB storage, $4.50/million requests
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| No egress fees | Newer service |
|
|
| Global CDN included | Fewer regions than S3 |
|
|
| S3-compatible API | Less tooling |
|
|
| Workers integration | |
|
|
|
|
#### Cost Estimate (10K apps, 100GB)
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| Storage (100GB) | $1.50 |
|
|
| Requests (1M) | $4.50 |
|
|
| Egress | $0 |
|
|
| **Total** | **~$6** |
|
|
|
|
---
|
|
|
|
### Option B: AWS S3 + CloudFront
|
|
|
|
```
|
|
Storage: S3 Standard
|
|
CDN: CloudFront
|
|
Egress: $0.085-0.12/GB (varies by region)
|
|
Pricing: $0.023/GB storage
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| Most mature | Egress costs add up |
|
|
| Best tooling | Complex pricing |
|
|
| All regions | Need CloudFront config |
|
|
| IAM integration | |
|
|
|
|
#### Cost Estimate (10K apps, 100GB, 1TB egress)
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| Storage (100GB) | $2.30 |
|
|
| CloudFront (1TB) | $85 |
|
|
| Requests | ~$5 |
|
|
| **Total** | **~$92** |
|
|
|
|
---
|
|
|
|
### Option C: Backblaze B2 + Cloudflare
|
|
|
|
```
|
|
Storage: Backblaze B2
|
|
CDN: Cloudflare (free egress via Bandwidth Alliance)
|
|
Egress: FREE (through Cloudflare)
|
|
Pricing: $0.005/GB storage
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| Cheapest storage | Two services to manage |
|
|
| Free egress via CF | B2 API slightly different |
|
|
| Good reliability | Need CF proxy setup |
|
|
|
|
#### Cost Estimate (10K apps, 100GB)
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| Storage (100GB) | $0.50 |
|
|
| Egress (via CF) | $0 |
|
|
| Requests | ~$0.40 |
|
|
| **Total** | **~$1** |
|
|
|
|
---
|
|
|
|
### Option D: Self-hosted MinIO
|
|
|
|
```
|
|
Storage: MinIO on VPS
|
|
CDN: Cloudflare proxy
|
|
Egress: VPS bandwidth
|
|
Pricing: VPS cost only
|
|
```
|
|
|
|
| Pros | Cons |
|
|
|------|------|
|
|
| Full control | Ops overhead |
|
|
| S3-compatible | Need to manage |
|
|
| Predictable cost | Scaling complexity |
|
|
|
|
#### Cost Estimate
|
|
|
|
| Component | Monthly Cost |
|
|
|-----------|--------------|
|
|
| VPS (500GB SSD) | $20-50 |
|
|
| Cloudflare | $0 (free tier) |
|
|
| **Total** | **~$20-50** |
|
|
|
|
---
|
|
|
|
## Recommendation
|
|
|
|
**Primary: Cloudflare R2**
|
|
- Zero egress fees (biggest cost saver)
|
|
- S3-compatible (easy migration)
|
|
- Built-in global CDN
|
|
- Good enough tooling
|
|
|
|
**Fallback: Backblaze B2 + Cloudflare**
|
|
- If R2 has issues
|
|
- Even cheaper storage
|
|
- Slightly more setup
|
|
|
|
---
|
|
|
|
## Upload Flow
|
|
|
|
### Presigned URL Approach
|
|
|
|
```
|
|
┌────────┐ ┌─────────┐ ┌─────────┐
|
|
│ Client │───►│ API │───►│ R2 │
|
|
└────────┘ └─────────┘ └─────────┘
|
|
│ │ │
|
|
│ 1. Request upload URL │
|
|
│◄─────────────┤ │
|
|
│ (presigned URL) │
|
|
│ │
|
|
│ 2. Upload directly │
|
|
│────────────────────────────►│
|
|
│ │
|
|
│ 3. Notify complete │
|
|
│─────────────►│ │
|
|
│ │ 4. Validate │
|
|
│ │─────────────►│
|
|
│ │ │
|
|
│ 5. Confirm │ │
|
|
│◄─────────────┤ │
|
|
```
|
|
|
|
### API Implementation
|
|
|
|
```go
|
|
// 1. Request upload URL
|
|
func CreateVersion(c *gin.Context) {
|
|
// Create version record
|
|
version := Version{
|
|
AppID: appID,
|
|
VersionCode: req.VersionCode,
|
|
Status: "uploading",
|
|
}
|
|
db.Create(&version)
|
|
|
|
// Generate presigned URL
|
|
key := fmt.Sprintf("temp/%s/package.mosis", version.ID)
|
|
url, err := r2.PresignPut(key, 15*time.Minute)
|
|
|
|
c.JSON(200, gin.H{
|
|
"version": version,
|
|
"upload_url": url,
|
|
"expires": time.Now().Add(15 * time.Minute),
|
|
})
|
|
}
|
|
|
|
// 3. Upload complete notification
|
|
func UploadComplete(c *gin.Context) {
|
|
// Move from temp to final location
|
|
tempKey := fmt.Sprintf("temp/%s/package.mosis", versionID)
|
|
finalKey := fmt.Sprintf("packages/%s/%s/%d/package.mosis",
|
|
developerID, appID, versionCode)
|
|
|
|
r2.Copy(tempKey, finalKey)
|
|
r2.Delete(tempKey)
|
|
|
|
// Update version status
|
|
version.Status = "validating"
|
|
version.PackageURL = finalKey
|
|
|
|
// Trigger async validation
|
|
queue.Publish("validate-package", version.ID)
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Download Flow
|
|
|
|
### Public Downloads
|
|
|
|
```go
|
|
// Get download URL (short-lived)
|
|
func GetDownloadURL(c *gin.Context) {
|
|
version := getLatestPublishedVersion(packageID)
|
|
|
|
// Generate presigned download URL (1 hour)
|
|
url, _ := r2.PresignGet(version.PackageURL, 1*time.Hour)
|
|
|
|
c.JSON(200, gin.H{
|
|
"download_url": url,
|
|
"version": version.VersionName,
|
|
"size": version.PackageSize,
|
|
"signature": version.Signature,
|
|
})
|
|
}
|
|
```
|
|
|
|
### CDN Caching
|
|
|
|
```
|
|
Cache-Control: public, max-age=86400
|
|
```
|
|
|
|
- Packages are immutable (version code = unique)
|
|
- Cache aggressively at edge
|
|
- Invalidate only if package is pulled
|
|
|
|
---
|
|
|
|
## Icon/Screenshot Handling
|
|
|
|
### Upload
|
|
|
|
```go
|
|
// Icons uploaded with app creation/update
|
|
func UploadIcon(c *gin.Context) {
|
|
file, _ := c.FormFile("icon")
|
|
|
|
// Validate dimensions
|
|
img, _ := png.Decode(file)
|
|
if img.Bounds().Dx() != img.Bounds().Dy() {
|
|
return error("Icon must be square")
|
|
}
|
|
|
|
// Generate multiple sizes
|
|
sizes := []int{32, 64, 128}
|
|
for _, size := range sizes {
|
|
resized := resize(img, size, size)
|
|
key := fmt.Sprintf("assets/%s/icon-%d.png", appID, size)
|
|
r2.Put(key, resized)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Serving
|
|
|
|
```
|
|
https://cdn.mosis.dev/assets/{app_id}/icon-64.png
|
|
```
|
|
|
|
- Public read access
|
|
- Long cache TTL (icons rarely change)
|
|
- Cloudflare image optimization (optional)
|
|
|
|
---
|
|
|
|
## Retention Policy
|
|
|
|
### Packages
|
|
|
|
| Status | Retention |
|
|
|--------|-----------|
|
|
| Published (current) | Forever |
|
|
| Published (old) | 1 year |
|
|
| Draft | 30 days |
|
|
| Rejected | 7 days |
|
|
| Failed validation | 24 hours |
|
|
|
|
### Temp Uploads
|
|
|
|
- Delete after 1 hour if not completed
|
|
- Lifecycle rule on `/temp/` prefix
|
|
|
|
### Implementation
|
|
|
|
```yaml
|
|
# R2 Lifecycle Rules
|
|
rules:
|
|
- prefix: "temp/"
|
|
expiration_days: 1
|
|
|
|
- prefix: "packages/"
|
|
tags:
|
|
status: draft
|
|
expiration_days: 30
|
|
```
|
|
|
|
---
|
|
|
|
## Backup Strategy
|
|
|
|
### R2 Built-in
|
|
|
|
- 11 nines durability
|
|
- Automatic replication within region
|
|
- No additional backup needed for packages
|
|
|
|
### Metadata Backup
|
|
|
|
- Package metadata in PostgreSQL
|
|
- PostgreSQL backups cover this
|
|
- Can regenerate URLs from DB
|
|
|
|
---
|
|
|
|
## Monitoring
|
|
|
|
### Metrics to Track
|
|
|
|
| Metric | Source |
|
|
|--------|--------|
|
|
| Upload success rate | API logs |
|
|
| Download latency | Cloudflare analytics |
|
|
| Storage usage | R2 dashboard |
|
|
| Bandwidth | R2 dashboard |
|
|
| Error rate | API logs |
|
|
|
|
### Alerts
|
|
|
|
- Upload failure rate > 5%
|
|
- Download error rate > 1%
|
|
- Storage > 80% of budget
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
### Access Control
|
|
|
|
```
|
|
Packages:
|
|
- Write: API only (presigned URLs)
|
|
- Read: Public (presigned URLs)
|
|
|
|
Assets:
|
|
- Write: API only
|
|
- Read: Public (direct CDN)
|
|
|
|
Temp:
|
|
- Write: Presigned URLs (15 min)
|
|
- Read: None (API internal only)
|
|
```
|
|
|
|
### Signed URLs
|
|
|
|
```go
|
|
// Presigned URL with expiration
|
|
url := r2.PresignPut(key, 15*time.Minute, PutOptions{
|
|
ContentType: "application/octet-stream",
|
|
ContentLength: maxSize,
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
## Deliverables
|
|
|
|
- [ ] R2 bucket setup
|
|
- [ ] Presigned URL generation
|
|
- [ ] Upload flow implementation
|
|
- [ ] Download URL generation
|
|
- [ ] Icon/screenshot upload
|
|
- [ ] Lifecycle rules for cleanup
|
|
- [ ] Monitoring dashboard
|
|
- [ ] Cost tracking
|
|
|
|
---
|
|
|
|
## Open Questions
|
|
|
|
1. Multi-region storage for lower latency?
|
|
2. Package compression (gzip/brotli)?
|
|
3. Delta updates storage structure?
|
|
4. Screenshot requirements (dimensions, count)?
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/)
|
|
- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
|
|
- [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html)
|