face-sets/docs/analysis/osrc-identity-discovery.md

Identity discovery in /mnt/x/src/osrc

Run date: 2026-04-26. Cache: work/cache/nl_full.npz (5260 face records). Driver script: work/cluster_osrc.py.

1. Source

/mnt/x/src/osrc/ is a flat mixed-identity bucket: 213 files in root + a psd/ subfolder with 41 PSD files + a single file in [Originaldateien]/. File extensions are 171 jpg + 1 jpeg + 41 psd. PSDs are not embedded (InsightFace's loader doesn't read PSD); the 41 PSDs were skipped, on the working assumption that the same identities are also present in the adjacent JPGs.

nl_full.npz already covered 160 of the 213 files (the remaining 53: 41 psd + 12 jpg). Of the 12 missing JPGs, 11 are byte-duplicates of 00843resc.jpg .. 00855resc.jpg (same file sizes, paired by sha256) — already aliased in the cache. Only 1 jpg (19554226_..._n.jpg) is genuinely uncovered.

The 160 covered files yielded 336 face records / 10 noface, with 64 single-face / 35 two-face / 19 three-face / 24 four-face / 8 with 5–8 faces. Quality is good: median face_short=116px, det_score=0.85, blur=244. Min face_short=40px will fail the 90px refine gate.

2. Coverage by existing identities

Computed cos-dist from each osrc face to the centroids of the canonical faceset_001..019 (built from each manifest's (source, bbox) keys). Median nearest-cos-dist was 0.875 — i.e. the bulk of osrc is not the existing 19 identities.

At cos-dist ≤ 0.45 (matching build_folders.py's OSRC_THRESHOLD):

existing identity	osrc faces matched
faceset_002	7
faceset_008	4
faceset_015	3
faceset_019	4

These 18 osrc faces are routed to existing identities by build_folders.py and extend; they are excluded from the identity-discovery step.

3. Pipeline

work/cluster_osrc.py mirrors build_folders.py's structure (synthesize a refine manifest, hand off to cmd_export_swap, relocate, merge top-level manifest) but discovers identities by clustering rather than asserting them by folder.

1. Filter cache to face records under /mnt/x/src/osrc (canonical or byte-aliased path).
2. Drop the 18 already-covered faces (cos-dist ≤ 0.45 to any existing identity centroid).
3. Cluster the remaining 318 faces among themselves at cos-dist 0.55 (matches the extend default for new-cluster formation).
4. For each cluster, apply refine-equivalent per-face gates (face_short ≥ 90, blur ≥ 40, det_score ≥ 0.6); for clusters ≥ 4 faces apply outlier rejection at cluster-centroid cos-dist 0.55. Keep clusters whose surviving unique-path count is ≥ 6 (the operator- chosen MIN_FACES, lower than the canonical 15 because osrc is small per-identity).
5. Number kept clusters faceset_020+ (past the existing facesets_swap_ready/ max of 019) ordered by size descending.
6. Synthesize a refine manifest and call cmd_export_swap on it. Move the emitted dirs into facesets_swap_ready/, drop an osrc.txt provenance marker, and append the new entries to the top-level manifest.json (without disturbing existing facesets / thin_eras).

4. Result (2026-04-26)

Phase 1 (clustering, before export-swap):

• 137 raw clusters at cos-dist 0.55; top sizes [37, 20, 12, 9, 7, 7, 6, 6, 6, 5].
• After quality gate: 124 faces dropped (mostly face_short < 90 from group-photo tertiary subjects).
• Outlier rejection: 0 dropped (clusters were tight).
• After min_faces=6: 7 candidate clusters kept (sizes 6–28 unique source paths).

Phase 2 (cmd_export_swap with min_face_short=100, outlier_threshold=0.45):

name	input	outlier drop	exported PNGs
faceset_020	71	42	26
faceset_021	36	21	10
faceset_022	15	7	8
faceset_023	19	14	4
faceset_024	6	0	6
faceset_025	10	4	6
faceset_026	—	—	0 (skipped: empty after filter)

faceset_026's 6 cluster faces all failed export-swap's tighter min_face_short=100 gate (vs. cluster's 90); it is not emitted. faceset_023 is small (4 PNGs) but useful as an averaged identity at that size.

Top-level facesets_swap_ready/manifest.json now: 31 substantive facesets (12 auto-cluster nl/lzbkp + 7 hand-sorted + 6 era splits + 6 osrc-discovered) + 68 thin_eras under _thin/.

5. Re-running and applying to other mixed buckets

• The cache holds osrc embeddings; to re-run with different parameters, edit cluster_osrc.py's config block and re-execute. Cluster discovery
  • export-swap is a few minutes total.
• For a different mixed-bucket source, copy cluster_osrc.py to cluster_<name>.py and change OSRC_DIR, OUT_TMP, SYNTH_MANIFEST, START_NNN. The exclusion step compares against the current contents of facesets_swap_ready/faceset_NNN/ so it picks up everything emitted by previous discovery / split / hand-sorted runs.
• Lowering MIN_FACES from 6 to 4 would have admitted ~3 additional marginal clusters at this corpus size; the trade-off is a noisier identity average for small-N facesets.
• extend should be run before cluster_osrc.py so raw_full/ and facesets_full/ stay in sync — cluster_osrc.py itself only writes to facesets_swap_ready/.