GRID) — forward spec¶

Status. The canvas-media data model and enums are fully in place, and the single-canvas main-video spine (SINGLE canvases + CANVAS_FILL edits) is live across upload, analysis, and render. Multi-panel composition — SPLIT, TALKING_HEAD, and GRID — is not yet built in the render pipeline or the structural editor. This document is the design for that deferred work. The base data model is documented in docs/backend/models.md; this doc covers only the composition layer on top of it.

INSERT canvases (full-frame asset between canvases) and project-spanning overlays/watermark/background-audio are already exercised by the b-roll path; the parts still to build are the dual-panel/PIP canvas types and the canvas-walking render dispatch.

Mental model¶

Three layers, each owning a distinct kind of statement — this is the rule for where a new property belongs:

Project — the render output as a whole: output resolution, default aspect ratio, audio normalization/denoise, caption track config, video flip. (Adding HDR touches Project.)
Canvas — composition of the frame during one source-time range: type, aspect override (composition-only), letterbox color, crop region, type-specific params. (Adding a new scene type touches Canvas.)
Edit — a media placement/modification/annotation within a canvas (or project-wide): role, transforms, audio mix, time range, action, source trim (media_offset_ms). (Adding chroma key touches Edit.)

Two invariants make this work:

Dual-POV. The same Edit row can be presented as "an asset chip over the waveform" (editing POV) or "a canvas in the source sequence" (sequence POV). The stored model is canvas-in-sequence; the UI may render either. Converting between "asset" and "clip" presentation is a role + canvas-membership change, not a type change.
Canvas activity is derived, not stored. A canvas has no active flag. Whether it renders as its declared type depends on whether its driving Edits are active in the current draft: if a SPLIT canvas's PANEL_B edit is toggled off, the renderer falls back to SINGLE for that range. This is what lets drafts adapt composition with no canvas_overrides field — the primary requirement the earlier "Layout" attempt failed.

Hard tiling¶

Every region of project source-time belongs to exactly one non-INSERT canvas — including SINGLE. INSERT canvases are zero-width (start_ms == end_ms) and sit between adjacent canvases at a source-time point, contributing output time without consuming source time.

Adding a SPLIT/TALKING_HEAD at [t1, t2] splits the covering canvas: SINGLE [0,t1] + SPLIT [t1,t2] + SINGLE [t2,end], where the two SINGLE neighbors reference the same MediaItem with different media_offset_ms windows. Deleting a composed canvas converts it back to SINGLE in place (it does not merge neighbors); merging two adjacent SINGLEs is an explicit gesture. Tiling ("non-INSERT canvases cover source-time exactly once, no gaps/overlaps; INSERTs are zero-width at boundaries") is enforced in CanvasService on create/update/delete, with neighbor re-densification of sequence_position in one transaction (the (project_id, sequence_position) unique constraint is DEFERRABLE INITIALLY DEFERRED for exactly this).

Canvas types¶

Canvas.type ∈ SINGLE | SPLIT | TALKING_HEAD | INSERT | GRID. Type-specific parameters live in Canvas.params (JSON), validated per-type by Pydantic (required fields enforced, extras rejected).

SINGLE (live) — one source fills the frame; driving edit role = CANVAS_FILL. Covers what were "clips" and REPLACE assets.
SPLIT (deferred) — two panels: edits role = PANEL_A and role = PANEL_B. params: split_ratio (0.1–0.9, default 0.5), split_direction (auto/horizontal/vertical; auto stacks by output aspect), split_invert (bool). Each panel letterboxes to its slot using letterbox_color. Both panels contribute audio via each edit's audio_volume_percent + audio_duck_main; default mix clip-side 100% / asset-side 0%.
TALKING_HEAD (deferred) — asset fills the canvas (role = CANVAS_FILL), the clip becomes a PIP bubble (role = PIP_SOURCE). params: pip_x/pip_y (0–1 center), pip_size_percent (10–50), pip_shape (rect/rounded/circle), pip_rotation_deg, pip_flip_h/pip_flip_v, and pip_crop_* (matching the per-asset crop machinery on Edit). Default audio: asset 100%, PIP clip 0%.
INSERT (live via b-roll) — full-frame asset spliced between canvases; start_ms == end_ms, one CANVAS_FILL edit whose output_duration_ms (non-NULL by constraint) is the output contribution.
GRID (reserved) — enum value exists, no implementation; reserves the schema for future N-panel grids.

Composition-relevant model additions¶

Beyond the base model (docs/backend/models.md):

Edit.role names the slot in a composition: CANVAS_FILL, PANEL_A, PANEL_B, PIP_SOURCE, OVERLAY, CUT, MUTE, BACKGROUND_AUDIO, WATERMARK. action (CUT/MUTE/KEEP) governs render behavior; role is categorical. Placements are action = KEEP; modifications are action ∈ {CUT, MUTE}.
CUTs are always canvas-level (a cut changes duration, so both panels of a SPLIT must cut together); MUTEs may be panel-targeted (role = PANEL_A/PANEL_B) since the mix is already per-panel.
output_duration_ms rule — for content-bearing edits (CANVAS_FILL/PANEL_A/PANEL_B/PIP_SOURCE/OVERLAY/BACKGROUND_AUDIO/WATERMARK), output contribution = output_duration_ms IF NOT NULL ELSE end_ms - start_ms. NULL for everything except INSERT CANVAS_FILL (where source span is zero and the field is required). Does not apply to CUT/MUTE. Forward-compatible with v2 playback-speed (source span vs output span diverge).
Project-spanning edits carry canvas_id = NULL: role = OVERLAY (project-wide decoration), WATERMARK (user-uploaded; the system/free-tier watermark stays render-time-synthesized with no row), BACKGROUND_AUDIO. INTRO/OUTRO are not roles — they are INSERT canvases at the lowest / highest sequence_position.
ProjectExport.canvas_snapshot — frozen canvas rows keyed by UUID at export time (Convention #12). Keying by UUID is load-bearing: the renderer does O(1) canvas_id → canvas lookups while interpreting each edit's coordinate space (below).

Render dispatch (canvas-walking, single FFmpeg pass)¶

Today ApplyCutsStep dispatches to one of three FFmpeg builders (broll two-cursor concat / overlay composite / simple trim), none canvas-aware. The composition path replaces that with one canvas-walking pass:

Load canvas_snapshot (dict by canvas_id), edit_snapshot, settings_snapshot. Output dims stay project-global (concat requires uniform dims).
Walk canvases in (project_id, sequence_position) order (hard tiling ⇒ full coverage) into (start_ms, end_ms, canvas, output_duration_ms) segments.
Per segment: resolve canvas-local cuts/mutes, apply canvas-local crop_region (or project default), build the canvas-type-specific composition (SINGLE fill / SPLIT two panels / TALKING_HEAD fill+PIP), apply canvas-local OVERLAY edits.
concat all pieces → apply project-spanning overlays + watermark → burn captions (project-global timeline) → audio: canvas-level amix per edit, then project-level normalize/denoise on the final concat.

Coordinate disambiguation is a correctness requirement. A single Edit.start_ms/end_ms now carries two coordinate systems: for a non-INSERT parent canvas the timestamps are project source-time; for an INSERT parent they are INSERT-internal output-time in [0, output_duration_ms]. Any render path reading a CUT/MUTE edit's timestamps MUST first resolve the parent type via canvas_snapshot[edit.canvas_id].type — hence the UUID-keyed snapshot. Interpreting a cut without that lookup mis-trims INSERT content.

Single FFmpeg invocation (not per-canvas encode + concat) avoids quality loss and GOP/boundary artifacts; the existing build_replace_filter_complex already expresses "route stream A for 0–10s, B for 10–15s, A again for 15–end" via concat, and the canvas-aware path generalizes it. Per-canvas crop_region and aspect_ratio_override are composition-only (content composed at the override aspect, then letterboxed/cropped to the fixed project output).

Audio lane — derived now, editable later¶

The composition layer above describes the visual lane. Audio is a second, derived lane and must not be modelled as a property of tiles.

v1 (ships with the canvas-walking render): every content-bearing element declares an audio contribution policy — SELF / UNDERLYING / NONE / MIX, plus gain and duck — derived from today's existing fields (audio_mode, audio_volume_percent, audio_duck_main, panel-targeted MUTEs, BACKGROUND_AUDIO). A pure function derive_audio_lane() resolves those into a flat list of AudioElements in output time, and the FFmpeg audio graph is built exclusively from that list. No schema change; no new columns.

Why audio is derived, not structural. A REPLACE b-roll is precisely the case where the two lanes diverge — b-roll picture, main-video audio. A tile model cannot express it: promoting the b-roll to a canvas fill destroys the reference to the main video's audio window for that span. This is why b-roll stays an instruction rather than being promoted to a tile, and why the underlying canvas fill continuing to span the range is what makes UNDERLYING resolvable for free.

Taxonomy consequence: REPLACE may remain an instruction permanently — it is intrinsically a visual-lane element. Converting placements into tiles applies to INSERT / SPLIT / TALKING_HEAD, not to REPLACE.

Deferred to v2 (do not build now) — file alongside playback-speed; same species of problem:

audio_element table — the derived IR plus uuid, project_id, detached BOOL. Materialise-on-first-touch, per element: a user's first direct edit persists that element as detached=true; everything untouched stays linked and keeps re-deriving. Detached wins on overlap.
Anchoring model (the open question) — are detached elements anchored to output time (absolute; cuts can break sync, as in real NLEs) or to a structural element (ripples with cuts, but needs a reference that survives the mutation workflows)? Same class as the rejected canvas_id-on-run-scoped-rows problem.
Lockstep rule becomes policy — today "a CUT hits both lanes" is an invariant; with detachment it becomes per-element behaviour.

Analysis over composed canvases¶

Analysis builds its source-time spine from the current canvas sequence (load_canvas_time_ranges), not any persisted recording order, so reorders/splits write through to the next run. INSERT canvases don't shift the source-time spine (zero width) but their audio is in the transcription stream, so transcript-time and canvas-source-time differ; analysis maps a transcript timestamp T back to a canvas by walking the assembled segments and subtracting accumulated INSERT output-durations (non-INSERT hit → project source-time; INSERT hit → INSERT-internal output-time). Analysis emits run-scoped edits and never creates/mutates canvas rows — structure is the editor's job.

Frontend composition + preview¶

Preview must stay WYSIWYG with export. Today VideoWindow renders two sibling layers — a cropped main-video layer and an asset layer (see the crop-layer split invariant in docs/development/frontend.md) — and there is no multi-panel preview component; an earlier project-wide Layout attempt that would have supplied one was abandoned before it shipped, so no LayoutComposedCanvas exists. The structural-editor work is therefore to build the composed-canvas preview and drive it off Canvas rows plus conversion gestures (create/split/merge/reorder), wiring it to the canvas-walking render path so preview and export share one composition source of truth.

Guarded until the dispatch exists. load_canvas_spine_for_render filters to SINGLE, so a SPLIT/TALKING_HEAD canvas would be dropped from the export without a trace. Two constants in modules/canvas/constants.py hold that shut from both ends: RENDERABLE_CANVAS_TYPES (SINGLE, INSERT) and DEFERRED_CANVAS_TYPES (everything else). create_layout_canvas refuses to create a deferred type, and the spine raises ValidationError rather than render around one that already exists.

Moving a type from DEFERRED_CANVAS_TYPES to RENDERABLE_CANVAS_TYPES opens creation and rendering together — there is no way to enable one without the other. Build the dispatch below first, then move the type; test_every_creatable_layout_type_is_currently_deferred fails if the two tuples drift apart.