Context and Transcripts Improve Detection of Deepfake Audios of Public Figures

TL;DR

This paper demonstrates that incorporating context and transcripts significantly enhances deepfake audio detection accuracy, introduces a novel architecture called CADD, and provides a new dataset for public figure deepfakes.

Contribution

The paper introduces CADD, a context-based audio deepfake detector, and a new dataset of public figure deepfakes, showing improved robustness and performance over existing methods.

Findings

Context and transcripts improve detection efficacy by up to 37.58% in F1-score.

CADD is more robust to adversarial evasion strategies, with less than 1% performance degradation.

Performance gains are consistent across multiple datasets and baseline detectors.

Abstract

Humans use context to assess the veracity of information. However, current audio deepfake detectors only analyze the audio file without considering either context or transcripts. We create and analyze a Journalist-provided Deepfake Dataset (JDD) of 255 public deepfakes which were primarily contributed by over 70 journalists since early 2024. We also generate a synthetic audio dataset (SYN) of dead public figures and propose a novel Context-based Audio Deepfake Detector (CADD) architecture. In addition, we evaluate performance on two large-scale datasets: ITW and P$^2$V. We show that sufficient context and/or the transcript can significantly improve the efficacy of audio deepfake detectors. Performance (measured via F1 score, AUC, and EER) of multiple baseline audio deepfake detectors and traditional classifiers can be improved by 5%-37.58% in F1-score, 3.77%-42.79% in AUC, and 6.17%-47.83% in EER. We additionally show that CADD, via its use of context and/or transcripts, is more robust to 5 adversarial evasion strategies, limiting performance degradation to an average of just -0.71% across all experiments. Code, models, and datasets are available at our project page: https://sites.northwestern.edu/nsail/cadd-context-based-audio-deepfake-detection (access restricted during review).