Download Pipeline¶

When users upload a video, import from YouTube, or add b-roll, we process the source into a unified MediaFile: content-addressable storage, extracted audio, a waveform, a thumbnail, and (for non-web-compatible video) a 480p proxy. Every source — main video, b-roll, image, audio — flows through the same worker and lands in one bucket (STORAGE_BUCKET_MEDIA).

The pipeline lives in backend/src/workers/download/media_pipeline/ and runs on two brokers: download_broker for the light artifacts (audio, waveform, thumbnail) and proxy_broker for the CPU-heavy 480p re-encode.

Two Entry Points¶

YouTube Import¶

POST /api/v1/media/youtube
{ "url": "https://youtube.com/watch?v=...", "scope": "PROJECT", ... }

Admin-only for now (get_current_superuser) while the feature waits on a bot-safe download server. The service fetches metadata, creates the MediaFile + MediaItem at PROCESSING, and — after commit — kicks download_youtube_media. Convention #16: the service manages its own sessions around the metadata fetch; the download itself runs in the worker. Convention #13 applies — the duration cap is enforced against resolved video metadata, not URL shape alone.

Presigned Upload (single-PUT, files < 25 MiB)¶

POST /api/v1/media/presign
{ "filename": "video.mp4", "content_type": "video/mp4", "size_bytes": ..., "scope": "PROJECT", ... }

Returns a presigned URL. The client PUTs the file body directly to R2, then confirms:

POST /api/v1/media/{media_file_id}/confirm

Confirm HEADs the object, re-checks quota against the actual Content-Length, then queues process_media_artifacts. This is the main confirm-path task.

Multipart Upload (files ≥ 25 MiB)¶

For files at or above MULTIPART_CUTOFF_BYTES (25 MiB), the client uses the multipart endpoints under /media/multipart/:

POST   /media/multipart/initiate     → upload_id + initial 50 part URLs
PUT    <part URL> (× N parts in parallel)
POST   /media/multipart/complete     → finalize, kick process_media_artifacts
POST   /media/multipart/abort        → cancel + release R2 parts
GET    /media/multipart/parts        → list parts already on R2 (resume)
GET    /media/multipart/parts/urls   → refill URLs as upload progresses

The complete endpoint queues process_media_artifacts, mirroring the single-PUT confirm path. See docs/backend/api.md Multipart Upload sections for full request/response schemas.

MediaFile Lifecycle¶

MediaFileStatus has six states. The load-bearing distinction: UPLOADED means "streamable + analysis-ready", not "fully processed". The 480p proxy is a non-blocking post-UPLOADED upgrade, tracked by the derived proxy_status (not_applicable / pending / ready / unknown), never a gate.

• Single-PUT / multipart upload: PENDING → PROCESSING → UPLOADED (or FAILED). Confirm/complete flips to PROCESSING and kicks process_media_artifacts, which finalizes to UPLOADED.
• YouTube import: PROCESSING → DOWNLOADING → PROCESSING → UPLOADED. download_youtube_media fetches into storage and hands off to process_media_artifacts.
• Multipart in-flight: MULTIPART_INITIATED. Rows still in this state are deleted on user cancel, complete-time over-quota, or cron sweep — never flipped to FAILED.

process_media_artifacts Task¶

The confirm-path worker (download_broker). Order is load-bearing:

1. Download from R2 — fetch the uploaded source into a temp dir.
2. sha256 hash — compute the content hash off the request path (asyncio.to_thread).
3. Content-addressable dedup — if the hash matches an existing UPLOADED MediaFile (the winner), repoint this row's MediaItems onto the winner, delete the loser row + its storage object, and short-circuit. Pre-check + UNIQUE-violation race-catch; the winner is read FOR UPDATE so status is ordered against the winner's own completion. See Content-Addressable Dedup.
4. Build artifacts (build_media_artifacts, dispatched by content type):
5. Video — probe codecs + dimensions + duration; extract audio (or synthesize silence for a silent source); generate waveform; generate thumbnail.
6. Audio — duration + waveform, no proxy.
7. Image — dimensions + thumbnail, no proxy.
8. Animated GIF — normalized to MP4 first (transcode_gif_to_mp4), then re-keyed and processed as an ordinary video. A single-frame GIF stays a static image.
9. Finalize — write the artifacts (storage_key, audio_key, waveform_json, has_audio, width, height, thumbnail_key, duration_ms, metadata_json) and flip to UPLOADED.
10. Seed canvases (seed_canvases_for_media_file) — grow the canvas spine for each PROJECT-scoped MediaItem backed by this file, so the analysis worker has structure to read. LIBRARY items seed nothing.
11. Emit media_ready (emit_media_ready) — one SSE event per referencing MediaItem, carrying the derived proxy_status.
12. Kick the proxy — if the source is non-web-compatible video, enqueue generate_media_proxy onto proxy_broker.

On the final retry attempt, an unrecoverable failure flips the row to FAILED and emits media_failed.

download_youtube_media Task¶

The YouTube analog of the single-PUT confirm. It only makes the MediaFile's storage_key valid, then hands off:

1. Download with yt-dlp — up to 1080p (MAX_VIDEO_HEIGHT), best audio, merged to MP4.
2. Upload to media storage — store in R2 under media/{prefix}/{uuid}/{title}.mp4.
3. set_media_source_downloaded — write the storage key and flip to PROCESSING.
4. Hand off — kick process_media_artifacts, which owns dedup, artifacts, canvas seeding, media_ready, and the proxy.

Idempotent: an already-UPLOADED row (a re-import that deduped onto it at the endpoint) skips the download and just re-seeds + notifies for the newly added MediaItem.

generate_media_proxy Task¶

Runs on the dedicated proxy_broker (queue: proxy), executed by the taskiq-proxy-worker container. CPU-heavy re-encodes can take 1-3× source duration, so decoupling this broker from download_broker keeps proxy generation off the analysis-feeding queue.

Creates a web-compatible preview plus a timeline scrub sprite:

1. Download original — from STORAGE_BUCKET_MEDIA.
2. Transcode + sprite — chained FFmpeg single-decode pass: 480p H.264 + AAC audio (with +faststart and dense keyframes for ~2s seek granularity) plus a 10×20 grid sprite (JPEG) from the same -i. -map 0:a? (optional) so audio-less inputs survive the encode.
3. Upload both — proxy to media/{prefix}/{uuid}/proxy.mp4, sprite to media/{prefix}/{uuid}/sprite.jpg.
4. set_media_proxy — set proxy_key, sprite_key, and sprite_seconds_per_tile; status stays UPLOADED.
5. Emit media_ready with proxy_status = ready.

A failure here does not flip the MediaFile to FAILED — it is already UPLOADED and analysis-ready; the proxy is non-blocking. taskiq retries; if it ultimately gives up, the row stays UPLOADED with proxy_status = pending and the stuck-proxy cron sweep recovers it. Sprite density (seconds_per_tile) is chosen at generation time as max(1, ceil(duration_s / 200)) — short clips get 1s/tile, long clips scale down so the sprite stays a fixed 10×20 grid.

Content-Addressable Dedup¶

MediaFile is content-addressable via sha256_hash (or youtube_video_id), under a UNIQUE constraint. When two uploads have the same bytes, the second becomes a dedup loser:

• Pre-check — an already-UPLOADED MediaFile with this hash exists. Repoint the loser's MediaItems onto it and delete the loser.
• Race-catch — claim_hash collides with a concurrent owner that may still be PROCESSING. Same repoint-onto-winner path; the winner is read FOR UPDATE.

The losing hash is never persisted (the UNIQUE constraint rejects the write), so a dedup loser never becomes a dedup target for a third upload — which is what makes inline-deleting it unconditionally safe. Quota is untouched: the loser's confirm-time per-MediaItem charge stands. The loser's storage object is deleted best-effort (infra-only, never touches quota).

Silent-Source Handling¶

MediaFile.has_audio is set by probing the source (audio_codec is None → False). Silent sources (iOS ReplayKit screen recordings, muted b-roll, PNGs, animated GIFs) get a synthesized silent WAV at audio_key so downstream consumers stay shape-stable, but the analysis pipeline routes Whisper / silence / false-start detection around them per-clip. The frontend disables the audio-dependent controls only when every source in the project is silent.

Canvas Seeding¶

Once an artifact finalize (or a dedup repoint onto an already-UPLOADED winner) makes a file analyzable, seed_canvases_for_media_file grows each referencing PROJECT's canvas spine — a SINGLE canvas + its CANVAS_FILL Edit per source — so the analysis worker has structure to read. Idempotent and zero-duration-safe via CanvasService.seed_single_canvas. This is the upload→canvas forward path; the editor owns all subsequent structural mutation.

Recovery¶

There is no per-source retry endpoint. A source that exits at FAILED is recovered by deleting it and re-uploading — a content-addressed re-upload of the same bytes deduplicates onto any surviving copy for free.

Stuck-row recovery is handled by the recover_stuck_tasks cron (_recover_stuck_media_files):

• PENDING past STUCK_MEDIA_FILE_MINUTES (abandoned single-PUT) → delete the MediaItem + MediaFile + R2 object.
• PROCESSING past STUCK_MEDIA_PROCESSING_MINUTES → FAILED.
• MULTIPART_INITIATED past the multipart window → abort on R2 (idempotent) + delete the rows.
• UPLOADED non-web-compatible with proxy_key IS NULL past STUCK_PROXY_MINUTES → FAILED.

Orphan reclamation runs in the daily cleanup_orphan_media_files cron: a MediaFile whose every MediaItem was deleted is reclaimed (row + R2 objects) after ORPHAN_GRACE_DAYS, race-safe under a per-row FOR UPDATE refcount lock.

Audio Extraction¶

We extract audio in Whisper-compatible format (16kHz mono WAV):

ffmpeg -i input.mp4 -vn -acodec pcm_s16le -ar 16000 -ac 1 output.wav

• -acodec pcm_s16le - uncompressed 16-bit PCM
• -ar 16000 - 16kHz sample rate (Whisper requirement)
• -ac 1 - mono channel

The audio is stored at audio_key and reused by the analysis pipeline. A silent source synthesizes an equivalent-duration silent WAV via anullsrc instead.

Waveform Generation¶

The waveform is an array of peak amplitudes used for the timeline visualization:

[0.12, 0.45, 0.78, 0.32, 0.91, ...]

We generate ~100 peaks per second of audio. The frontend renders these as vertical bars in the timeline. Stored on MediaFile.waveform_json.

Codec Compatibility¶

Web browsers can play:

• Video: H.264, VP8, VP9
• Audio: AAC, MP3, Opus

If we detect non-compatible codecs (HEVC, ProRes, AV1 in some browsers, animated GIF), we generate a proxy for preview via generate_media_proxy. The original stays intact for rendering. Web-compatible sources skip the proxy entirely (proxy_status = not_applicable).

Storage Keys¶

All artifacts live under one bucket (STORAGE_BUCKET_MEDIA) and one key layout:

media/{prefix}/{uuid}/source              # Original upload / download
media/{prefix}/{uuid}/audio.wav           # Extracted (or synthesized) audio
media/{prefix}/{uuid}/proxy.mp4           # Web-compatible proxy
media/{prefix}/{uuid}/sprite.jpg          # Timeline scrub sprite (10x20 grid)
media/{prefix}/{uuid}/thumbnail.jpg       # Thumbnail
media/{prefix}/{uuid}/asset.mp4           # GIF-normalized MP4 (re-keyed)

The {prefix} is the first 2 characters of the UUID, which helps R2 distribute keys across partitions. Keys are minted by generate_media_storage_key.

Key Files¶

← Render Pipeline Analysis Pipeline →