face-sets

Sort photos by similar face using InsightFace embeddings + agglomerative clustering, refine into per-identity sets, and export ready-to-drop bundles for face-swap tooling (roop-unleashed, etc.).

Pipeline

sort_faces.py is a single-file CLI with six subcommands:

step	what it does
embed	Recursively scan a source tree, detect + embed every face, write .npz cache. Resumable; sha256-dedup.
cluster	Raw agglomerative clustering of the cache into person_NNN/ / _singletons/ / _noface/ with manifest.
refine	Initial cluster → centroid merge → quality gate → outlier rejection → size filter → faceset_NNN/.
dedup	Post-hoc near-duplicate report: byte-identical + visual near-dupe groups → <cache>.duplicates.json.
extend	Fold new embeddings into an existing raw/refine output via nearest person-centroid without renumbering.
enrich	Re-detect each cached face to persist landmark_2d_106, landmark_3d_68, pose (pitch/yaw/roll) into cache.
export-swap	Per-identity export: tight outlier gate + visual-dupe collapse + composite quality rank + single-face PNG crops + .fsz bundles (top-N and full) ready for roop-unleashed. Optional singleton rescue into _candidates/.

Design principles

• embed is resumable and incremental. It loads any existing cache at the target path and only hashes / embeds files it has not seen. Atomic flush every 50 new files so a mid-run crash loses at most ~50 embeddings.
• Byte-identical duplicates are sha256-grouped at listing time. The canonical file is embedded once; other paths with the same hash become path_aliases in the cache. Every alias is materialized by cluster / refine / export-swap, so each on-disk location is represented.
• safe_dst_name always flattens the absolute path. This keeps output filenames stable across runs even as src_root changes between embed / extend / export invocations.
• Caches and outputs stay out of git via .gitignore; defaults live under work/.

Typical end-to-end run

SRC=/mnt/x/src/nl
CACHE=work/cache/nl_full.npz
OUT=/mnt/e/temp_things/fcswp/nl_sorted

# 1. Embed (CPU; InsightFace buffalo_l). Resumable on re-run.
python sort_faces.py embed "$SRC" "$CACHE"

# 2. Raw clusters (one person_NNN/ per multi-face cluster).
python sort_faces.py cluster "$CACHE" "$OUT/raw_full"

# 3. Refined facesets (quality-gated per-identity sets).
python sort_faces.py refine  "$CACHE" "$OUT/facesets_full"

# 4. Near-duplicate report (byte + visual).
python sort_faces.py dedup   "$CACHE"

# 5. Enrich the cache with landmarks + pose (needed by export-swap).
python sort_faces.py enrich  "$CACHE"

# 6. Export roop-unleashed-ready bundles.
python sort_faces.py export-swap "$CACHE" \
  "$OUT/facesets_full/refine_manifest.json" "$OUT/facesets_swap_ready" \
  --raw-manifest "$OUT/raw_full/manifest.json" --candidates

Merging a new source into an existing result

# Embed new source into the same cache (resume from existing embeddings + aliases).
python sort_faces.py embed /mnt/x/src/lzbkp_red "$CACHE"

# Fold new faces into raw_full + facesets_full without renumbering.
python sort_faces.py extend "$CACHE" "$OUT/raw_full" --refine-out "$OUT/facesets_full"

# Refresh the swap-ready export to reflect the merge.
python sort_faces.py enrich "$CACHE"
python sort_faces.py export-swap "$CACHE" \
  "$OUT/facesets_full/refine_manifest.json" "$OUT/facesets_swap_ready" \
  --raw-manifest "$OUT/raw_full/manifest.json" --candidates

Key defaults

refine:

flag	default	meaning
--initial-threshold	0.55	cosine distance for stage-1 clustering
--merge-threshold	0.40	centroid-level merge of over-split clusters
--outlier-threshold	0.55	drop face if cosine dist from centroid exceeds (only if cluster ≥ 4)
--min-faces	15	minimum unique images per faceset
--min-short	90	minimum short-edge pixels of face bbox
--min-blur	40.0	Laplacian-variance blur gate
--min-det-score	0.6	InsightFace detector score gate

export-swap:

flag	default	meaning
--top-n	30	size of the <faceset>_topN.fsz bundle
--outlier-threshold	0.45	tighter than refine; trims cluster boundary for averaging
--pad-ratio	0.5	padding around face bbox for PNG crop
--out-size	512	PNG output is square out_size × out_size
--min-face-short	100	export gate; stricter than refine's 90
--candidates	off	rescue _singletons/ into _candidates/ for manual review
--candidate-match-threshold	0.55	cos-dist cutoff for singleton → existing faceset
--candidate-min-score	0.40	composite-quality floor for candidates

The composite quality score in export-swap is 0.30·frontality + 0.20·det_score + 0.20·landmark_symmetry + 0.15·face_size + 0.15·sharpness, each normalized to [0, 1].

Downstream: roop-unleashed

The .fsz bundles emitted by export-swap drop straight into roop-unleashed's Face Swap tab. Each PNG inside is already a clean single-face crop — critical, because the roop-unleashed loader appends every face it re-detects in each PNG to the averaged identity embedding.

Highly recommended at swap time: enable Select post-processing = GFPGAN with the Original/Enhanced image blend ratio = 0.85 (default is 0.65 which is conservative). See docs/analysis/facesets-downstream-refinement-evaluation.md for the full evaluation.

Layout

/opt/face-sets/
├─ README.md                                     (this file)
├─ sort_faces.py                                 (the tool)
├─ docs/
│  └─ analysis/
│     └─ facesets-downstream-refinement-evaluation.md
└─ work/                                         (gitignored)
   ├─ cache/
   │  └─ nl_full.npz                             (canonical cache + duplicates.json)
   └─ logs/
      └─ *.log                                   (every long step writes here)