dataset forensics

DOLOS (Guo et al., ICCV 2023) is the largest audio-visual deception dataset: 1,675 clips of panel-show statements, each labeled truth or lie, distributed as YouTube links plus timestamps. That distribution model rots. By 2026, rights-holder takedowns had destroyed ~12% of the source uploads, the timestamp manifest carried spreadsheet-conversion damage, and every published benchmark number rested on cross-validation folds that quietly recycle the same people between train and test.

Before this corpus could anchor any training claim, it needed an audit — and then a rebuild.

• Manifest corruption, four modes — 44 rows of Excel damage: start/end columns holding (end, duration) pairs, decimal truncation misreading 27:20 as 27:02, minute-digit typos, and one clip name pointing at two different moments. Each mode was reconstructed from internal evidence — annotation intervals that must fall inside their clip windows act as ground truth.
• Identity leakage, 96–99% — at the person level, nearly every test clip in the official folds comes from a speaker seen in training. Published accuracy on this benchmark measures person recognition as much as deception detection. An honest protocol requires person-disjoint splits, with recurring hosts pinned to the training side.
• Confounded labels — lie rates differ by gender, several behavioral markers reverse direction across gender groups (a live Simpson's paradox), and a handful of frequent panelists carry extreme lie-rate profiles. Any model can shortcut through identity unless the split design forbids it.

Higher-quality broadcast sources exist for the underlying episodes, but the official timestamps don't transfer: source uploads were compilations with cold-opens, mid-episode edits, and re-cuts. The rebuild aligns purely by audio — 100 Hz loudness envelopes, matched-filter cross-correlation of each original clip against every candidate episode — and then refuses to trust itself:

• Content gate — every re-cut clip's audio must correlate near-unity with the original clip's own audio. Loud regions (credits, applause) false-lock naive correlation; windowed normalization kills the artifact.
• Window gate — content match is not position match. A cut offset by two seconds still correlates on the overlap; the acceptance test also bounds the residual alignment shift.
• Decoy controls — every model-assisted verification step carries planted negatives. One "clean" sweep was exposed as a stopped clock this way: the verifying model was silently receiving frames without audio and answering no to everything, including the decoys.

Result: 98% of the achievable corpus recovered at broadcast quality, every clip carrying a named verification tier, and a documented ledger of what remains unverifiable and why. One misaligned episode was caught after the fact by an annotation model describing applause where a statement should be — the disagreement signal doubling as a cut validator.

Benchmarks decay: link rot, conversion damage, and leakage accumulate silently while papers keep citing the original numbers. The pattern here generalizes — treat the dataset itself as an artifact under test, verify every transformation against ground truth the artifact already contains, and design each verification step so it is able to fail.