face-sets/docs/analysis/video-target-preprocessing.md

Video target preprocessing for roop-unleashed

Initial design + first batch run: 2026-04-27. Driver scripts: work/video_target_pipeline.py, work/video_face_worker.py, work/run_video_pipeline.sh.

Companion to the face-set side of the project: instead of building per-identity .fsz bundles for the source of a swap, this pipeline preprocesses the target (videos to swap into). Given a folder of video files, it identifies "swappable" segments — continuous shots where a face is detectable, sufficiently visible, and roughly within inswapper_128's working envelope — and cuts them into UUID-named clips ready to feed into roop-unleashed.

1. Why build it

I checked the obvious open-source projects for an existing implementation:

• FaceFusion (github.com/facefusion/facefusion) — CLI has run, headless-run, batch-run, job-*, force-download, benchmark. No scene-detection or clip-extraction subcommand. Its own guides recommend "split your video manually first."
• roop-unleashed at /opt/roop-unleashed/roop/util_ffmpeg.py — has cut_video(start_frame, end_frame) for a manual GUI trim, no detection-driven segmentation.
• Deep-Live-Cam (github.com/hacksider/Deep-Live-Cam) — real-time / single-shot, no batch preprocessing.
• DeepFaceLab — extract_video.bat dumps every frame between user-supplied trim points; no quality gating.

Closest prior art for the cut-detection pattern is the two-stage hybrid in SportSBD MMSys'26 (cheap detector for cuts, accurate net for verification), but the actual implementation has to be ours.

2. Pipeline architecture

WSL  /opt/face-sets/work/                   Windows  C:\face_embed_venv\
─────────────────────────────────────       ─────────────────────────────
run_video_pipeline.sh (chain driver)
   │
   ├─ scan         (ffprobe metadata)
   ├─ scenes       (PySceneDetect AdaptiveDetector, CPU)
   ├─ stage        (sampled frame queue.json @ 2 fps)
   │                                  │
   │                                  ▼
   │                            video_face_worker.py
   │                            insightface FaceAnalysis
   │                            on DmlExecutionProvider
   │                            output: results.jsonl
   ├─ merge        (ingest results.jsonl)
   ├─ track        (IoU + embedding stitching, ~30 LOC)
   ├─ score        (track-level quality gate + cross-track merge)
   ├─ cut          (ffmpeg -c copy → per-source subfolders)
   └─ report       (HTML preview)

   Output: <output_dir>/<source_video_stem>/<uuid>.mp4
                                           /<uuid>.json (sidecar; opt-in via
                                                          --write-sidecar)

run_video_pipeline.sh is parameterized via env vars (WORK, INPUT_DIR, OUTPUT_DIR, FILTER_FROM, SKIP_PATTERN, MAX_DUR, IDENTITY, SIDECAR) so you can pin a particular batch without editing the script. Sidecars are off by default — the per-batch plan.json always carries the full provenance for every clip; the <uuid>.json files alongside the clips are redundant and only useful if you need each clip to be self-describing in isolation.

3. Quality signals (matched to inswapper_128's working envelope)

inswapper_128 is trained near-frontal at 128×128. The score gate uses defaults that admit side profiles (since rich face-sets can absorb non-frontal swap targets):

signal	threshold	rationale
`	yaw	`
`	pitch	`
face_short	≥ 80 px	inswapper resamples to 128; ≥80 still produces clean output
det_score	≥ 0.5	matches buffalo_l's MIN_DET; lower = unreliable detection
track-gate	≥ 70 % frames pass	binary track filter rather than per-frame
duration	1 s ≤ dur ≤ 120 s	below 1s = unusable slivers; above 120s probably contains a missed micro-cut

Plus two segment-merging knobs:

• --bridge-gap (default 3 s) — within a single track, brief pose-failure gaps shorter than this get bridged so single bad frames don't fragment a good run
• --merge-gap (default 2 s) — across tracks within the same scene, segments closer than this get fused (cross-track merge fires when face detection briefly fails between adjacent good runs)

The defaults can be tightened (e.g. --max-yaw 25 for portrait-only) or loosened (e.g. --max-yaw 90 --merge-gap 5) without re-running detection — score reads the existing tracks.json.

4. Performance + the JSONL append-only fix

This is where the engineering interest is. The first production run on 13 videos / 6.18 h of input went through three failure modes before settling at production speed:

attempt	issue	rate observed
1. Original cap.set(POS_FRAMES, N) per sample	OpenCV seeks to nearest keyframe + decodes forward at every sample. Cost grows with depth into the video; on a 60-min H.264 it falls off a cliff.	1.4 fps → degrading
2. Sequential cap.grab() from frame 0	On resume, grab-walking from frame 0 to a deep target is unbounded.	0.08 fps
3. Hybrid: seek-once-per-video + sequential within	Better in principle. But hit a different bug: flush() was re-serializing the entire results.json (245 MB at this point) every 100 frames or 30 sec. Save dominated wall-clock.	0.5 fps
4. JSONL append-only	One result per line. Each flush is O(new records), not O(total records).	13.77 fps smoke / 7.57 fps cumulative across the full batch

Lesson: when the output is large + grows monotonically + needs frequent checkpointing, do not re-serialize the whole structure on each flush. Append-only line-delimited JSON is the right tool. The legacy results.json is auto-converted to .jsonl on first load (one-time migration), so resumes survive the format switch.

5. Hardware decode/encode on AMD Vega + WSL

Skipped. Per Microsoft's WSL D3D12 video acceleration post, VAAPI-via-Mesa-D3D12 exists but is fragile on older AMD. AMF on Windows would mean a Windows-side ffmpeg leg, doubling boundary crossings. CPU software decode of 1280×720 H.264 in WSL ffmpeg is faster than realtime, and the bottleneck is buffalo_l detection on DML, not decode.

For cutting we use -c copy stream-copy — no re-encode, hardware codecs are moot.

6. Full corpus run results

Three runs across the 61-video corpus at /mnt/x/src/vd/:

	test (3 videos)	first batch (13 videos, 50–62)	rest (45 videos, 02–49 minus test)	total
input duration	0.6 h	6.18 h	12.98 h	19.76 h
sampled frames @ 2 fps	4,472	44,635	94,030	143,137
tracks	187	2,564	3,823	6,574
accepted tracks	94 (50 %)	1,193 (47 %)	1,905 (50 %)	3,192 (49 %)
emitted segments	83	600	1,301	1,984
cross-track-merged segments	14	254	382	650
accepted content	13 min	239 min	395 min	647 min (= 10.78 h)
acceptance rate by time	36 %	64.6 %	50.7 %	54.6 %
output size	0.135 GB	3.63 GB	4.84 GB	8.6 GB

Phase timings (rest batch — best representative since it ran fully under JSONL append-only from a fresh start):

• scenes: 117 min (PySceneDetect, 45 × ~3 min/video)
• stage: instant
• worker: 100 min @ 15.78 fps sustained (vs 7.5 fps for first batch which migrated mid-run)
• merge: 90 s
• track: 92 s
• score: 23 s
• cut (1,301 ffmpeg stream-copies): 30 min
• report (1,301 thumbs + HTML): 5.5 min
• total wall-clock: 4h16m

Across all three runs, 0 worker errors on 143,137 sampled frames.

7. Re-running

# choose a per-batch workdir + log
WORK=/opt/face-sets/work/video_preprocess_<batch_name> \
  FILTER_FROM=ct_src_00050.mp4 \
  bash work/run_video_pipeline.sh > work/logs/video_run_<batch_name>.log 2>&1 &

# check status anytime
bash work/status_video_pipeline.sh work/logs/video_run_<batch_name>.log

Skip patterns can exclude already-processed inputs (note that 5-digit numbers need full padding in the regex, e.g. 0005[0-9] not 005[0-9]):

SKIP_PATTERN='^ct_src_(0001[015]|0005[0-9]|0006[0-2])\.mp4$' \
  WORK=/opt/face-sets/work/video_preprocess_rest \
  bash work/run_video_pipeline.sh > work/logs/video_run_rest.log 2>&1 &

To also emit per-clip provenance sidecars (off by default):

SIDECAR=yes \
  WORK=/opt/face-sets/work/video_preprocess_<batch> \
  bash work/run_video_pipeline.sh > work/logs/video_run_<batch>.log 2>&1 &

scenes outputs are cached in the batch's WORK/scenes/ dir, so re-running the chain after an edit-to-score step doesn't redo detection. The worker is also resumable per queue_id — if killed mid-flight, just relaunch.