Sapari Infrastructure Architecture¶

Context¶

Sapari is an AI-powered video editing platform that automates post-production for content creators: silence removal, false start detection, profanity filtering, asset compositing, and caption generation. Users interact with a browser-based editor (React 19) backed by a FastAPI API, with heavy processing handled by background workers consuming from RabbitMQ priority queues via TaskIQ.

The infrastructure progresses through versions as load grows. v1.0 is the launch architecture optimized for cost. v2.0 is the endgame with GPU acceleration and Kubernetes autoscaling. Intermediate versions split the workload incrementally.

Current version: v1.0 (launch)

Design Principles¶

1. Scale vertically as long as possible. A bigger box is operationally cheaper than a cluster.
2. No hard lock-in. Every external service has a migration path. Workers are Docker containers that only need queue access.
3. Colocate compute and I/O. Workers, Redis, and temporary video storage live on the same Hetzner machine (or same datacenter at v1.2+) for low latency.
4. Minimize ops surface. Plain Docker Compose at v1.x. Move to k3s only when horizontal scaling forces it.
5. Defer expensive infrastructure. GPU server (€184/mo) is justified only when transcription costs on the OpenAI API exceed the GPU's amortized cost.

Version Progression¶

Each version is a fully working state. You can stay at any version indefinitely.

v1.0: Launch Architecture (current)¶

Overview¶

graph TB
    subgraph Cloudflare
        CF_PAGES["Cloudflare Pages<br/><i>Frontend (React 19) + Landing (Astro)</i>"]
        CF_R2["R2 Storage<br/><i>media (unified) + exports buckets</i>"]
        CF_DNS["DNS + CDN"]
        CF_WORKER["Workers<br/><i>/api/* proxy to backend</i>"]
        CF_ACCESS["Access<br/><i>Staging gate (GitHub OAuth)</i>"]
    end

    subgraph Neon
        POSTGRES_PROD[("PostgreSQL<br/><i>sapari-production project</i>")]
        POSTGRES_STG[("PostgreSQL<br/><i>sapari-staging project</i>")]
    end

    subgraph "Production Server (Hetzner, 16 vCPU / 32 GB / 640 GB)"
        CADDY_P["Caddy 2 (TLS via DNS-01)"]
        API_P["FastAPI Backend"]
        WORKERS_P["6 Workers + Scheduler<br/><i>analysis, render, download, proxy, media-edit, email</i>"]
        REDIS_P[("Redis 7<br/><i>cache + sessions + pub/sub</i>")]
        RMQ_P[("RabbitMQ 3<br/><i>task broker + management UI</i>")]
    end

    subgraph "Staging Server (Hetzner, 4 vCPU / 8 GB / 160 GB)"
        CADDY_S["Caddy 2"]
        API_S["FastAPI Backend"]
        WORKERS_S["6 Workers + Scheduler"]
        REDIS_S[("Redis 7")]
        RMQ_S[("RabbitMQ 3")]
    end

    %% User flows -- production
    USER["User Browser"] -- "HTTPS" --> CF_DNS
    CF_DNS --> CF_PAGES
    CF_DNS --> CF_WORKER
    CF_WORKER -- "/api/*" --> CADDY_P
    CF_WORKER -- "static" --> CF_PAGES
    CADDY_P --> API_P
    API_P --> POSTGRES_PROD
    API_P -- "presigned" --> CF_R2
    API_P --> REDIS_P
    API_P --> RMQ_P
    RMQ_P --> WORKERS_P
    WORKERS_P --> POSTGRES_PROD
    WORKERS_P --> CF_R2
    WORKERS_P --> REDIS_P

    %% Staging gated by Access
    DEV["Developer"] -- "GitHub login" --> CF_ACCESS
    CF_ACCESS --> CADDY_S
    CADDY_S --> API_S

Physical Layout¶

Managed Services¶

Networking¶

graph LR
    USER["User Browser"]

    subgraph "Cloudflare Edge"
        WORKER["Worker proxy"]
        PAGES["Pages (static assets)"]
    end

    subgraph "Hetzner Server (Production)"
        CADDY["Caddy<br/>api.sapari.io<br/>(TLS via DNS-01)"]
        BACKEND["Backend container"]
    end

    USER -- "https://app.sapari.io" --> WORKER
    WORKER -- "/api/* (HTTPS)" --> CADDY
    WORKER -- "static" --> PAGES
    CADDY --> BACKEND

Why Cloudflare Worker proxy: Frontend hardcodes relative /api/v1 path (shared/api/client.ts:7) and cookies use SameSite=Strict in production (auth/session/manager.py:18,523). Cross-origin would require code changes + weakening CSRF posture. The Worker makes frontend and backend appear same-origin to the browser. Backend cookies are host-only (no domain= set, verified manager.py:526-544), so they bind naturally to the Worker's domain.

Why DNS-only (gray cloud) on api.sapari.io: Browser never sees this domain. Worker fetches it directly. Adding Cloudflare proxy would double-proxy (CF -> CF -> Caddy -> Backend), adding latency.

TLS strategy: Caddy uses Cloudflare DNS-01 ACME challenge (no port 80 exposed). UFW only allows 443 publicly + 22 from operator IP. Production additionally locks the Docker DOCKER-USER chain so :443 inbound only accepts traffic from Cloudflare egress CIDRs — UFW alone doesn't cover Docker-NAT'd ports because Docker manages its own iptables rules that run before UFW's INPUT chain. The lockdown is applied by sapari-docker-firewall.service (a oneshot systemd unit installed by setup-server.sh); it fetches Cloudflare's published v4 + v6 egress lists at boot, scopes the rules to the WAN interface, and re-applies idempotently on every restart. Direct hits to the public IP on :443 from non-CF sources refuse the connection.

Stack on Each Server¶

Single Docker Compose file (docker-compose.prod.yml), shared between staging and production with different env files:

Single image, multiple commands: All backend containers run from the same Docker image. They differ only in the command (fastapi run vs taskiq worker analysis_broker etc.). Trade-off: code changes affect all containers, but build time stays low and operations stay simple.

Deployment¶

GitHub Actions builds + pushes Docker images to GHCR on push to staging or main. Server SSH is firewalled to the tailnet only, so deploy workflows run tailscale/github-action@v3 to join the tailnet before the appleboy/ssh-action step, then execute ./scripts/deployment/deploy.sh on the server.

CD branch flow: - staging branch -> Build -> auto-deploy to staging server. The sapari-frontend-staging Pages project auto-deploys staging.sapari.io on the same push. - main branch -> Build only. Production deploys are manual via Actions -> Deploy Production -> Run workflow with typed YES confirmation. That workflow deploys the backend over SSH and then, once its health check passes, fires a Cloudflare Pages deploy hook to ship the frontend. The sapari-landing Pages project auto-deploys sapari.io from main.

Production uses a manual trigger (workflow_dispatch + typed confirm) rather than a required-reviewer gate, because private repos without GitHub Team can't use reviewer or wait-timer environment protection rules. The typed confirm is the deliberate-action equivalent.

The production frontend does not auto-deploy, on purpose. Automatic deployments are disabled on the sapari-frontend Pages project. Were they on, a staging -> main merge would ship the SPA immediately while the backend still waits for its manual dispatch — the frontend would lead the API it calls, every release. Routing the frontend through a deploy hook at the end of deploy-production.yml makes the ordering structural rather than a thing to remember. See Frontend deploys.

Deploy = stop-then-start per service. Brief downtime per service (seconds) during docker compose up -d. React Query retries + maintenance screen handle it.

Backups¶

No backup scripts in v1.0 -- everything is a managed-service toggle.

Cost Summary (v1.0)¶

Variable costs: - LLM API (DeepSeek + OpenAI Whisper + GPT-5.x for analysis steps) -- per-token - Postmark email -- $1.25 per 10K emails - Stripe -- 2.9% + 30¢ per transaction (standard) - R2 beyond 10 GB storage -- \(0.015/GB/mo - Neon Launch upgrade if 100 CU-hrs/mo exceeded -- ~\)20-30/mo per project

v1.1: Vertical Scale¶

Trigger: Production RAM consistently >75% during peak hours, or render queue regularly >2 deep. (Current headroom is large: compose CPU limits total 10.15 against 16 vCPU, memory limits ~11.8 GB against 32 GB.)

Change: Resize the Hetzner box in-place (1-2 minutes downtime during reboot). Same architecture, more resources.

What changes: - Increase TASKIQ_WORKER_CONCURRENCY from 1 to 2 or 3 in .env.production (more concurrent renders) - Increase mem_limit on workers in docker-compose.prod.yml (more headroom) - Optionally raise FFMPEG_THREADS if CPU is consistently underutilized

What doesn't change: Domain layout, Cloudflare config, services, deployment process. Just a bigger box.

v1.2: Horizontal Split (API + Workers)¶

Trigger: Workers and API contend for resources after vertical scaling is exhausted; worker OOM starts affecting API responsiveness. (Not close today — compose CPU limits total 10.15 against 16 vCPU, memory ~11.8 GB against 32 GB.)

Change: Move workers to a dedicated server. API stays on its own. WireGuard tunnel between them for private Redis + RabbitMQ access.

graph TB
    subgraph "API Server (illustrative)"
        CADDY_API["Caddy"]
        API["FastAPI"]
    end

    subgraph "Worker Server (illustrative, larger)"
        WORKERS["6 workers + scheduler"]
        REDIS[("Redis")]
        RMQ[("RabbitMQ")]
    end

    USER -- "https://app.sapari.io" --> CF["Cloudflare Worker proxy"]
    CF --> CADDY_API
    CADDY_API --> API
    API -- "WireGuard 10.0.0.0/24" --> REDIS
    API -- "WireGuard" --> RMQ
    WORKERS --> REDIS
    WORKERS --> RMQ
    API --> NEON[("Neon Postgres")]
    WORKERS --> NEON
    WORKERS --> R2[("R2")]
    API --> R2

Cost example: the SKU names and prices in this future-architecture section were written against the stale v1.0 baseline (a 4 vCPU / 16 GB prod box) and have not been re-quoted. Treat them as illustrative shape, not as current figures.

What changes: - Provision second Hetzner box, set up WireGuard between API box and worker box - API box .env.production points Redis/RabbitMQ to 10.0.0.2 (worker box's WireGuard IP) - Workers move to the new box, API stays on the old one - Backup Redis + RabbitMQ data on the worker box (Hetzner backups still cover both) - DNS unchanged (api.sapari.io still points to API box)

What doesn't change: Cloudflare layer, Neon, R2, deployment scripts (just deploy to two servers instead of one).

v1.3: Managed Data Plane¶

Trigger: Self-hosting Redis and RabbitMQ becomes operational burden. RabbitMQ broker config drifts, Redis memory pressure causes evictions, you've spent more time tuning them than the value they provide.

Change: Replace self-hosted Redis with Upstash, RabbitMQ with CloudAMQP. WireGuard tunnel goes away (back to public TLS for everything).

What changes: - Both servers connect to managed Redis/RabbitMQ over public TLS - Server budget shrinks (no need for the worker box's RAM to host Redis/RabbitMQ) - WireGuard goes away - Backup story simplifies (managed services back themselves up)

What doesn't change: Cloudflare layer, Neon, R2, application code (env vars point at new endpoints).

v2.0: GPU + Kubernetes¶

Trigger: EITHER (a) worker box capacity exceeded even with managed data plane, OR (b) OpenAI Whisper API costs exceed ~$200/mo (~33,000 minutes of audio at $0.006/min).

Change: Full architecture from the original infrastructure plan. GEX44 GPU server runs analysis (local Whisper) + render (NVENC) + media-edit. CCX13 control plane runs k3s + remaining workers + managed RMQ/Redis (or keep managed). KEDA autoscales render workers per queue depth. Burst CCX13 nodes spin up on demand.

v2.0 Architecture (full)¶

This was the original infrastructure plan. Preserving it as the v2.0 endgame:

Physical Infrastructure¶

All Hetzner resources in HIL (Hillsboro, US-West) or ASH (Ashburn, US-East) datacenter, vSwitch private network (10.0.0.0/24).

k3s Cluster¶

graph TB
    SERVER["k3s Server (CCX13 #1)<br/>Control plane + etcd"]
    AGENT1["k3s Agent (GEX44)<br/>Labels: gpu=true, role=worker"]
    AGENT2["k3s Agent (CCX13 #2)<br/>Labels: role=burst-render"]

    SERVER --> AGENT1
    SERVER --> AGENT2

KEDA Autoscaling¶

• Render worker: 0-6 replicas, 1 pod per pending task in queue, 5-min cooldown
• Analysis worker: 1-2 replicas (GPU shared via semaphore), scales when queue >3
• Media-edit worker: 0-3 replicas, 2 tasks per pod, 2-min cooldown

When pending pods exceed GEX44 capacity, cluster-autoscaler provisions a CCX13 burst node (~2-3 minute provisioning + k3s join time).

GPU Acceleration¶

The RTX 4000 has independent NVENC/NVDEC and CUDA engines, enabling concurrent Whisper transcription (CUDA) and FFmpeg rendering (NVENC). Render workers detect GPU at startup and use h264_nvenc; CPU-only nodes (burst) fall back to libx264.

Pluggable Transcription¶

TranscriptionRouter selects backends based on availability and cost: - LocalWhisperBackend (GPU): preferred, ~€0/min amortized, 1 concurrent (semaphore) - OpenAIWhisperBackend: overflow when GPU busy, $0.006/min, 5 concurrent - GroqWhisperBackend (future): $0.003/min if added

Networking¶

WireGuard tunnel from API host (still on Hetzner or moved to Render/FastAPI Cloud) to control plane. Internal Hetzner traffic uses vSwitch.

Migration from v1.x to v2.0¶

This is a substantial migration, not a toggle. Major steps: 1. Provision GEX44 + CCX13 #1 2. Install k3s, NVIDIA drivers, container toolkit 3. Deploy backend image + workers as k8s manifests (existing Dockerfile works) 4. Wire KEDA against existing RabbitMQ 5. DNS cutover for api.sapari.io (or split into separate subdomains per worker pool) 6. Decommission v1.x server

Not in scope until v2.0 trigger fires. Documented as the endgame, not the next step.

Observability (all versions)¶

• Logfire (free tier) -- structured logs + OpenTelemetry traces. FastAPI, SQLAlchemy, Redis, Pydantic AI all instrumented.
• Beszel (beszel-staging.sapari.io) -- system + container metrics with threshold alerts. Canonical post-#42.
• Dozzle (dozzle-staging.sapari.io) -- live container log tailing across every service.
• System Health admin page -- link landing pad pointing at Beszel/Dozzle/internal-docs. Overview tab carries a quick-glance health/queue snippet via /admin/audit/health + /admin/audit/queues.
• RabbitMQ Management UI (port 15672 via SSH tunnel) -- queue depth, message rates, per-priority breakdown.
• Hetzner Cloud console -- CPU, RAM, disk graphs.
• Sentry (frontend only at v1.0) -- backend Sentry is a follow-up.

Migration Paths Summary¶

Open Questions¶

1. When to split API + workers (v1.0 -> v1.2)? Concrete trigger: 2+ concurrent renders queueing for >5 min during peak. Vague "feels slow" doesn't qualify.

2. When to add GPU (v1.x -> v2.0)? Math-based: when monthly OpenAI Whisper bill > Hetzner GEX44 cost (€184). At $0.006/min, that's ~30,500 minutes of audio per month -- equivalent to ~500 medium-length videos.

3. When to move backend off Hetzner self-host? Likely never. FastAPI Cloud might tempt, but custom domain support and pricing are still maturing.

4. NVENC quality vs CPU libx264? NVENC at preset p5 is faster but slightly lower quality. CPU libx264 produces smaller files at the same quality. v2.0 could offer "high quality" exports routed to CPU burst nodes.