AI Papers Reader

Personalized digests of latest AI research

View on GitHub

New Benchmark Shows Deepfake Detectors Struggle With Modern AI Video

A new benchmark called DF26 suggests that both people and leading deepfake detectors may have trouble telling modern AI-generated public-speaking videos from real footage. The problem matters because fabricated clips of officials, interviews, and direct-to-camera statements are plausible vehicles for disinformation, while many detectors are still tested on older face-swap datasets.

Researchers from Ukrainian Catholic University, Czech Technical University in Prague, Hover, and Ukraine’s National Academy of Sciences built DF26 from 271 real videos and 2,420 synthetic clips, for 2,691 videos total. The real clips come from public video sources and are divided into three scenarios: direct-to-camera/casual, official statements, and studio interviews. For each real video, the team used Gemini 2.5 to generate a semantic prompt from sampled frames. Open-source models were then used in text-to-video (T2V) and, where supported, image-to-video (I2V) modes; commercial systems were used only in T2V mode. The seven generators are Wan 2.2 A14B, HunyuanVideo 1.5, and LTX 2.3 distilled (open source), plus Kling 3.0, Veo 3.1, Wan 2.6, and Grok Imagine 1.0 (commercial). Because the prompt or first frame is matched to each real clip, the benchmark tries to isolate the effect of generator shift.

The paper reports that detectors trained on FaceForensics++ (FF++) degrade sharply on DF26. On CelebDF++ (CDFv3), temporal detector DFD-FCG reached 94.3 AUROC, a common ranking metric where 100 is perfect and 50 is chance; on DF26 it fell to 48.2, a 46.1-point drop. PwTF-DVD dropped from 92.3 to 61.6 AUROC, a 30.7-point decline. The best DF26 AUROC among evaluated detectors was 69.7 for GenD-PE. The variation across generators was stark: PwTF-DVD scored 92.9 AUROC on Wan 2.2 T2V but only 41.9 on HunyuanVideo 1.5 I2V, below chance. The authors note that aggregate scores can hide severe failures.

Human results pointed the same way. In a study with 232 labeling sessions, participants identified fake videos with 52.6% accuracy on DF26, near chance, versus 74.5% on CDFv3 and 69.8% on DeepSpeak v2. Accuracy on real videos was similar across datasets—76.0%, 75.5%, and 72.8%, respectively—so the difficulty was not that DF26’s real clips looked fake, the authors argue, but that participants could not find reliable evidence in the generated ones.

The study also found a gap between generator types. For two temporal detectors, AUROC on open-source generators was higher than on closed-source commercial generators: DFD-FCG scored 57.4 versus 34.5, and PwTF-DVD scored 70.5 versus 48.3. The authors call this diagnostic rather than causal, because source type may correlate with model family, post-processing, visual quality, or generation mode.

Important limits remain. DF26 contains 2,691 videos, small relative to large-scale benchmarks, and it evaluates only visual artifacts, not audio, speech quality, lip-sync, or audiovisual consistency. Commercial generators were evaluated only in T2V mode, and official-statement clips were excluded from commercial generation because of platform restrictions. The results therefore show a challenging benchmark, not a finished answer; the authors argue for cross-generator evaluation and detectors trained on newer data.