After completing the rest-of-corpus run, update docs/analysis to reflect
the final numbers across all three batches (test + 13-file + 45-file)
and surface the numerical lessons:
- 1,984 segments / 10.78h accepted content from 19.76h / 61 input videos
- 0 worker errors across 143,137 sampled frames
- rest batch sustained 15.78 fps from a fresh JSONL start (vs 7.5 fps for
the migrated batch), confirming the append-only fix is the right
steady-state design
- skip-pattern note: 5-digit basename numbers need full padding
(0005[0-9] not 005[0-9]) — bit me on the first relaunch
- documented SIDECAR=yes opt-in for the chain script
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously every video_target_pipeline cut wrote a <uuid>.json provenance
sidecar alongside each <uuid>.mp4. The same provenance is already in the
per-batch plan.json, so the per-clip sidecars are redundant unless a
downstream tool wants each clip self-describing in isolation.
- video_target_pipeline.py cut: new --write-sidecar flag, default off.
- run_video_pipeline.sh: new SIDECAR env var (default "no"), passes
--write-sidecar when SIDECAR=yes.
- README + docs/analysis/video-target-preprocessing.md updated.
The 1,984 already-emitted sidecars in /mnt/x/src/vd/ct/ct_src_*/ have
been deleted (1.5 MB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Preprocesses a folder of video files into UUID-named clips suitable as
target inputs for roop-unleashed-style face-swap. Counterpart to the
faceset (source-side) tooling.
work/video_target_pipeline.py — orchestration with subcommands
scan / scenes / stage / merge / track / score / cut / report. Quality
gates default to face-sets-can-handle-side-profile values (yaw<=75°,
pitch<=45°, face_short>=80px, det>=0.5). Cross-track segment merge
fuses adjacent-in-time tracks within the same scene up to 2s gap.
Output organized into <output_dir>/<source_stem>/<uuid>.mp4 +
<uuid>.json sidecar with full provenance.
work/video_face_worker.py — Windows DML face detect+embed worker. Uses
JSONL append-only for results.jsonl: a critical perf fix (re-
serializing the monolithic 245MB results.json on every flush was the
dominant cost in the first attempt, dropping throughput to 0.5 fps).
Append-only got it to 13+ fps, ~7.5 fps cumulative across the first
6.18h batch. Also uses seek-once-per-video + sequential cap.grab()
between samples to dodge cv2 per-sample seek pathology on long H.264.
Legacy results.json is auto-migrated to .jsonl on first load.
work/run_video_pipeline.sh — generic chain driver, parameterized via
WORK / INPUT_DIR / OUTPUT_DIR / FILTER_FROM / SKIP_PATTERN / MAX_DUR /
IDENTITY env vars. work/status_video_pipeline.sh — generic status
helper.
First production batch (ct_src_00050..00062, 13 files, 6.18h input):
600 emitted segments, 239.5min accepted content (64.6% of input), 254
segments built from >=2 tracks (cross-track merge), 1h43m wall clock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
