Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling

TL;DR

This paper introduces HBMNet, a hierarchical network for localizing deepfake regions in audio-visual content, effectively handling partial manipulations by integrating multi-scale cues and boundary-content relationships.

Contribution

The paper presents a novel Hierarchical Boundary Modeling Network that combines multi-scale temporal cues and boundary-content relationships for improved deepfake localization.

Findings

HBMNet outperforms existing methods BA-TFD and UMMAFormer.

Frame-level supervision enhances recall in localization.

Multi-scale and bidirectional modeling improve precision and overall performance.

Abstract

Audio-visual temporal deepfake localization under the content-driven partial manipulation remains a highly challenging task. In this scenario, the deepfake regions are usually only spanning a few frames, with the majority of the rest remaining identical to the original. To tackle this, we propose a Hierarchical Boundary Modeling Network (HBMNet), which includes three modules: an Audio-Visual Feature Encoder that extracts discriminative frame-level representations, a Coarse Proposal Generator that predicts candidate boundary regions, and a Fine-grained Probabilities Generator that refines these proposals using bidirectional boundary-content probabilities. From the modality perspective, we enhance audio-visual learning through dedicated encoding and fusion, reinforced by frame-level supervision to boost discriminability. From the temporal perspective, HBMNet integrates multi-scale cues and bidirectional boundary-content relationships. Experiments show that encoding and fusion primarily improve precision, while frame-level supervision boosts recall. Each module (audio-visual fusion, temporal scales, bi-directionality) contributes complementary benefits, collectively enhancing localization performance. HBMNet outperforms BA-TFD and UMMAFormer and shows improved potential scalability with more training data.