AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs

TL;DR

AVFakeBench is a new comprehensive benchmark for audio-video forgery detection that covers diverse forgery types and annotations, aiming to improve detection methods and evaluate large language models' capabilities.

Contribution

This paper introduces AVFakeBench, the first extensive benchmark for AV forgery detection with rich semantics and multi-task evaluation, addressing limitations of previous datasets.

Findings

AV-LMMs show potential as forgery detectors.

Existing methods struggle with fine-grained perception.

Benchmark reveals weaknesses in current detection approaches.

Abstract

The threat of Audio-Video (AV) forgery is rapidly evolving beyond human-centric deepfakes to include more diverse manipulations across complex natural scenes. However, existing benchmarks are still confined to DeepFake-based forgeries and single-granularity annotations, thus failing to capture the diversity and complexity of real-world forgery scenarios. To address this, we introduce AVFakeBench, the first comprehensive audio-video forgery detection benchmark that spans rich forgery semantics across both human subject and general subject. AVFakeBench comprises 12K carefully curated audio-video questions, covering seven forgery types and four levels of annotations. To ensure high-quality and diverse forgeries, we propose a multi-stage hybrid forgery framework that integrates proprietary models for task planning with expert generative models for precise manipulation. The benchmark establishes a multi-task evaluation framework covering binary judgment, forgery types classification, forgery detail selection, and explanatory reasoning. We evaluate 11 Audio-Video Large Language Models (AV-LMMs) and 2 prevalent detection methods on AVFakeBench, demonstrating the potential of AV-LMMs as emerging forgery detectors while revealing their notable weaknesses in fine-grained perception and reasoning.