Cross-Modal Watermarking for Authentic Audio Recovery and Tamper Localization in Synthesized Audiovisual Forgeries

TL;DR

This paper introduces a cross-modal watermarking framework that embeds authentic audio into visuals to enable recovery and tamper localization in synthesized audiovisual forgeries, enhancing misinformation detection and mitigation.

Contribution

The paper proposes a novel cross-modal watermarking approach for authentic audio recovery and tamper localization in SAVFs, addressing limitations of existing detection methods.

Findings

Effective in recovering authentic audio from manipulated content

Accurate tamper localization in audiovisual forgeries

Robust against voice cloning and lip synchronization manipulations

Abstract

Recent advances in voice cloning and lip synchronization models have enabled Synthesized Audiovisual Forgeries (SAVFs), where both audio and visuals are manipulated to mimic a target speaker. This significantly increases the risk of misinformation by making fake content seem real. To address this issue, existing methods detect or localize manipulations but cannot recover the authentic audio that conveys the semantic content of the message. This limitation reduces their effectiveness in combating audiovisual misinformation. In this work, we introduce the task of Authentic Audio Recovery (AAR) and Tamper Localization in Audio (TLA) from SAVFs and propose a cross-modal watermarking framework to embed authentic audio into visuals before manipulation. This enables AAR, TLA, and a robust defense against misinformation. Extensive experiments demonstrate the strong performance of our method in AAR and TLA against various manipulations, including voice cloning and lip synchronization.