Zero-Shot Fake Video Detection by Audio-Visual Consistency

TL;DR

This paper proposes a zero-shot fake video detection method based on audio-visual content consistency, utilizing pre-trained ASR and VSR models to compare content sequences and identify fakes without needing forged data.

Contribution

It introduces a novel zero-shot detection approach using content sequence comparison via edit distance, enhancing generalizability and robustness against various deepfake techniques.

Findings

Outperforms semantic and temporal consistency methods in generalization

Demonstrates robustness against audio-visual perturbations

Achieves state-of-the-art detection accuracy when combined with other systems

Abstract

Recent studies have advocated the detection of fake videos as a one-class detection task, predicated on the hypothesis that the consistency between audio and visual modalities of genuine data is more significant than that of fake data. This methodology, which solely relies on genuine audio-visual data while negating the need for forged counterparts, is thus delineated as a `zero-shot' detection paradigm. This paper introduces a novel zero-shot detection approach anchored in content consistency across audio and video. By employing pre-trained ASR and VSR models, we recognize the audio and video content sequences, respectively. Then, the edit distance between the two sequences is computed to assess whether the claimed video is genuine. Experimental results indicate that, compared to two mainstream approaches based on semantic consistency and temporal consistency, our approach achieves superior generalizability across various deepfake techniques and demonstrates strong robustness against audio-visual perturbations. Finally, state-of-the-art performance gains can be achieved by simply integrating the decision scores of these three systems.