add developer portal planning documentation (M01-M12)
This commit is contained in:
451
DEV_PORTAL_M07_STORAGE.md
Normal file
451
DEV_PORTAL_M07_STORAGE.md
Normal file
@@ -0,0 +1,451 @@
|
||||
# Milestone 7: CDN & Storage
|
||||
|
||||
**Status**: Planning
|
||||
**Goal**: Scalable storage for app packages and assets with global distribution.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Storage handles app packages, icons, screenshots, and serves downloads to devices worldwide. Must be cost-effective, fast, and reliable.
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
|
||||
- Store app packages (.mosis files)
|
||||
- Store icons (multiple sizes)
|
||||
- Store screenshots (optional)
|
||||
- Serve downloads globally
|
||||
- Support presigned upload URLs
|
||||
- Version retention (keep old versions)
|
||||
|
||||
### Non-Functional
|
||||
|
||||
| Requirement | Target |
|
||||
|-------------|--------|
|
||||
| Upload speed | < 30s for 50MB |
|
||||
| Download latency | < 100ms (p50) |
|
||||
| Availability | 99.9% |
|
||||
| Durability | 99.999999999% (11 nines) |
|
||||
| Cost | Minimize egress fees |
|
||||
|
||||
---
|
||||
|
||||
## Storage Structure
|
||||
|
||||
```
|
||||
/packages/
|
||||
/{developer_id}/
|
||||
/{app_id}/
|
||||
/{version_code}/
|
||||
package.mosis
|
||||
manifest.json (extracted for quick access)
|
||||
|
||||
/assets/
|
||||
/{app_id}/
|
||||
icon-32.png
|
||||
icon-64.png
|
||||
icon-128.png
|
||||
screenshots/
|
||||
1.png
|
||||
2.png
|
||||
3.png
|
||||
|
||||
/temp/
|
||||
/{upload_id}/
|
||||
package.mosis (pending validation)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Options Analysis
|
||||
|
||||
### Option A: Cloudflare R2
|
||||
|
||||
```
|
||||
Storage: Object storage (S3-compatible)
|
||||
CDN: Cloudflare network (automatic)
|
||||
Egress: FREE (zero egress fees)
|
||||
Pricing: $0.015/GB storage, $4.50/million requests
|
||||
```
|
||||
|
||||
| Pros | Cons |
|
||||
|------|------|
|
||||
| No egress fees | Newer service |
|
||||
| Global CDN included | Fewer regions than S3 |
|
||||
| S3-compatible API | Less tooling |
|
||||
| Workers integration | |
|
||||
|
||||
#### Cost Estimate (10K apps, 100GB)
|
||||
|
||||
| Component | Monthly Cost |
|
||||
|-----------|--------------|
|
||||
| Storage (100GB) | $1.50 |
|
||||
| Requests (1M) | $4.50 |
|
||||
| Egress | $0 |
|
||||
| **Total** | **~$6** |
|
||||
|
||||
---
|
||||
|
||||
### Option B: AWS S3 + CloudFront
|
||||
|
||||
```
|
||||
Storage: S3 Standard
|
||||
CDN: CloudFront
|
||||
Egress: $0.085-0.12/GB (varies by region)
|
||||
Pricing: $0.023/GB storage
|
||||
```
|
||||
|
||||
| Pros | Cons |
|
||||
|------|------|
|
||||
| Most mature | Egress costs add up |
|
||||
| Best tooling | Complex pricing |
|
||||
| All regions | Need CloudFront config |
|
||||
| IAM integration | |
|
||||
|
||||
#### Cost Estimate (10K apps, 100GB, 1TB egress)
|
||||
|
||||
| Component | Monthly Cost |
|
||||
|-----------|--------------|
|
||||
| Storage (100GB) | $2.30 |
|
||||
| CloudFront (1TB) | $85 |
|
||||
| Requests | ~$5 |
|
||||
| **Total** | **~$92** |
|
||||
|
||||
---
|
||||
|
||||
### Option C: Backblaze B2 + Cloudflare
|
||||
|
||||
```
|
||||
Storage: Backblaze B2
|
||||
CDN: Cloudflare (free egress via Bandwidth Alliance)
|
||||
Egress: FREE (through Cloudflare)
|
||||
Pricing: $0.005/GB storage
|
||||
```
|
||||
|
||||
| Pros | Cons |
|
||||
|------|------|
|
||||
| Cheapest storage | Two services to manage |
|
||||
| Free egress via CF | B2 API slightly different |
|
||||
| Good reliability | Need CF proxy setup |
|
||||
|
||||
#### Cost Estimate (10K apps, 100GB)
|
||||
|
||||
| Component | Monthly Cost |
|
||||
|-----------|--------------|
|
||||
| Storage (100GB) | $0.50 |
|
||||
| Egress (via CF) | $0 |
|
||||
| Requests | ~$0.40 |
|
||||
| **Total** | **~$1** |
|
||||
|
||||
---
|
||||
|
||||
### Option D: Self-hosted MinIO
|
||||
|
||||
```
|
||||
Storage: MinIO on VPS
|
||||
CDN: Cloudflare proxy
|
||||
Egress: VPS bandwidth
|
||||
Pricing: VPS cost only
|
||||
```
|
||||
|
||||
| Pros | Cons |
|
||||
|------|------|
|
||||
| Full control | Ops overhead |
|
||||
| S3-compatible | Need to manage |
|
||||
| Predictable cost | Scaling complexity |
|
||||
|
||||
#### Cost Estimate
|
||||
|
||||
| Component | Monthly Cost |
|
||||
|-----------|--------------|
|
||||
| VPS (500GB SSD) | $20-50 |
|
||||
| Cloudflare | $0 (free tier) |
|
||||
| **Total** | **~$20-50** |
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Primary: Cloudflare R2**
|
||||
- Zero egress fees (biggest cost saver)
|
||||
- S3-compatible (easy migration)
|
||||
- Built-in global CDN
|
||||
- Good enough tooling
|
||||
|
||||
**Fallback: Backblaze B2 + Cloudflare**
|
||||
- If R2 has issues
|
||||
- Even cheaper storage
|
||||
- Slightly more setup
|
||||
|
||||
---
|
||||
|
||||
## Upload Flow
|
||||
|
||||
### Presigned URL Approach
|
||||
|
||||
```
|
||||
┌────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Client │───►│ API │───►│ R2 │
|
||||
└────────┘ └─────────┘ └─────────┘
|
||||
│ │ │
|
||||
│ 1. Request upload URL │
|
||||
│◄─────────────┤ │
|
||||
│ (presigned URL) │
|
||||
│ │
|
||||
│ 2. Upload directly │
|
||||
│────────────────────────────►│
|
||||
│ │
|
||||
│ 3. Notify complete │
|
||||
│─────────────►│ │
|
||||
│ │ 4. Validate │
|
||||
│ │─────────────►│
|
||||
│ │ │
|
||||
│ 5. Confirm │ │
|
||||
│◄─────────────┤ │
|
||||
```
|
||||
|
||||
### API Implementation
|
||||
|
||||
```go
|
||||
// 1. Request upload URL
|
||||
func CreateVersion(c *gin.Context) {
|
||||
// Create version record
|
||||
version := Version{
|
||||
AppID: appID,
|
||||
VersionCode: req.VersionCode,
|
||||
Status: "uploading",
|
||||
}
|
||||
db.Create(&version)
|
||||
|
||||
// Generate presigned URL
|
||||
key := fmt.Sprintf("temp/%s/package.mosis", version.ID)
|
||||
url, err := r2.PresignPut(key, 15*time.Minute)
|
||||
|
||||
c.JSON(200, gin.H{
|
||||
"version": version,
|
||||
"upload_url": url,
|
||||
"expires": time.Now().Add(15 * time.Minute),
|
||||
})
|
||||
}
|
||||
|
||||
// 3. Upload complete notification
|
||||
func UploadComplete(c *gin.Context) {
|
||||
// Move from temp to final location
|
||||
tempKey := fmt.Sprintf("temp/%s/package.mosis", versionID)
|
||||
finalKey := fmt.Sprintf("packages/%s/%s/%d/package.mosis",
|
||||
developerID, appID, versionCode)
|
||||
|
||||
r2.Copy(tempKey, finalKey)
|
||||
r2.Delete(tempKey)
|
||||
|
||||
// Update version status
|
||||
version.Status = "validating"
|
||||
version.PackageURL = finalKey
|
||||
|
||||
// Trigger async validation
|
||||
queue.Publish("validate-package", version.ID)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Download Flow
|
||||
|
||||
### Public Downloads
|
||||
|
||||
```go
|
||||
// Get download URL (short-lived)
|
||||
func GetDownloadURL(c *gin.Context) {
|
||||
version := getLatestPublishedVersion(packageID)
|
||||
|
||||
// Generate presigned download URL (1 hour)
|
||||
url, _ := r2.PresignGet(version.PackageURL, 1*time.Hour)
|
||||
|
||||
c.JSON(200, gin.H{
|
||||
"download_url": url,
|
||||
"version": version.VersionName,
|
||||
"size": version.PackageSize,
|
||||
"signature": version.Signature,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### CDN Caching
|
||||
|
||||
```
|
||||
Cache-Control: public, max-age=86400
|
||||
```
|
||||
|
||||
- Packages are immutable (version code = unique)
|
||||
- Cache aggressively at edge
|
||||
- Invalidate only if package is pulled
|
||||
|
||||
---
|
||||
|
||||
## Icon/Screenshot Handling
|
||||
|
||||
### Upload
|
||||
|
||||
```go
|
||||
// Icons uploaded with app creation/update
|
||||
func UploadIcon(c *gin.Context) {
|
||||
file, _ := c.FormFile("icon")
|
||||
|
||||
// Validate dimensions
|
||||
img, _ := png.Decode(file)
|
||||
if img.Bounds().Dx() != img.Bounds().Dy() {
|
||||
return error("Icon must be square")
|
||||
}
|
||||
|
||||
// Generate multiple sizes
|
||||
sizes := []int{32, 64, 128}
|
||||
for _, size := range sizes {
|
||||
resized := resize(img, size, size)
|
||||
key := fmt.Sprintf("assets/%s/icon-%d.png", appID, size)
|
||||
r2.Put(key, resized)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Serving
|
||||
|
||||
```
|
||||
https://cdn.mosis.dev/assets/{app_id}/icon-64.png
|
||||
```
|
||||
|
||||
- Public read access
|
||||
- Long cache TTL (icons rarely change)
|
||||
- Cloudflare image optimization (optional)
|
||||
|
||||
---
|
||||
|
||||
## Retention Policy
|
||||
|
||||
### Packages
|
||||
|
||||
| Status | Retention |
|
||||
|--------|-----------|
|
||||
| Published (current) | Forever |
|
||||
| Published (old) | 1 year |
|
||||
| Draft | 30 days |
|
||||
| Rejected | 7 days |
|
||||
| Failed validation | 24 hours |
|
||||
|
||||
### Temp Uploads
|
||||
|
||||
- Delete after 1 hour if not completed
|
||||
- Lifecycle rule on `/temp/` prefix
|
||||
|
||||
### Implementation
|
||||
|
||||
```yaml
|
||||
# R2 Lifecycle Rules
|
||||
rules:
|
||||
- prefix: "temp/"
|
||||
expiration_days: 1
|
||||
|
||||
- prefix: "packages/"
|
||||
tags:
|
||||
status: draft
|
||||
expiration_days: 30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
### R2 Built-in
|
||||
|
||||
- 11 nines durability
|
||||
- Automatic replication within region
|
||||
- No additional backup needed for packages
|
||||
|
||||
### Metadata Backup
|
||||
|
||||
- Package metadata in PostgreSQL
|
||||
- PostgreSQL backups cover this
|
||||
- Can regenerate URLs from DB
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Metrics to Track
|
||||
|
||||
| Metric | Source |
|
||||
|--------|--------|
|
||||
| Upload success rate | API logs |
|
||||
| Download latency | Cloudflare analytics |
|
||||
| Storage usage | R2 dashboard |
|
||||
| Bandwidth | R2 dashboard |
|
||||
| Error rate | API logs |
|
||||
|
||||
### Alerts
|
||||
|
||||
- Upload failure rate > 5%
|
||||
- Download error rate > 1%
|
||||
- Storage > 80% of budget
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
### Access Control
|
||||
|
||||
```
|
||||
Packages:
|
||||
- Write: API only (presigned URLs)
|
||||
- Read: Public (presigned URLs)
|
||||
|
||||
Assets:
|
||||
- Write: API only
|
||||
- Read: Public (direct CDN)
|
||||
|
||||
Temp:
|
||||
- Write: Presigned URLs (15 min)
|
||||
- Read: None (API internal only)
|
||||
```
|
||||
|
||||
### Signed URLs
|
||||
|
||||
```go
|
||||
// Presigned URL with expiration
|
||||
url := r2.PresignPut(key, 15*time.Minute, PutOptions{
|
||||
ContentType: "application/octet-stream",
|
||||
ContentLength: maxSize,
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
- [ ] R2 bucket setup
|
||||
- [ ] Presigned URL generation
|
||||
- [ ] Upload flow implementation
|
||||
- [ ] Download URL generation
|
||||
- [ ] Icon/screenshot upload
|
||||
- [ ] Lifecycle rules for cleanup
|
||||
- [ ] Monitoring dashboard
|
||||
- [ ] Cost tracking
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Multi-region storage for lower latency?
|
||||
2. Package compression (gzip/brotli)?
|
||||
3. Delta updates storage structure?
|
||||
4. Screenshot requirements (dimensions, count)?
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/)
|
||||
- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
|
||||
- [Backblaze B2 + Cloudflare](https://www.backblaze.com/b2/docs/cloud_to_cloud.html)
|
||||
Reference in New Issue
Block a user