MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models

TL;DR

MARE leverages multimodal alignment and reinforcement learning with human feedback to improve deepfake detection accuracy and explainability using vision-language models, capturing intrinsic forgery traces.

Contribution

The paper introduces MARE, a novel framework combining multimodal alignment, reinforcement learning, and forgery disentanglement for enhanced explainable deepfake detection.

Findings

Achieves state-of-the-art accuracy in deepfake detection.

Provides explainable reasoning content aligned with human preferences.

Effectively captures intrinsic forgery traces from facial semantics.

Abstract

Deepfake detection is a widely researched topic that is crucial for combating the spread of malicious content, with existing methods mainly modeling the problem as classification or spatial localization. The rapid advancements in generative models impose new demands on Deepfake detection. In this paper, we propose multimodal alignment and reinforcement for explainable Deepfake detection via vision-language models, termed MARE, which aims to enhance the accuracy and reliability of Vision-Language Models (VLMs) in Deepfake detection and reasoning. Specifically, MARE designs comprehensive reward functions, incorporating reinforcement learning from human feedback (RLHF), to incentivize the generation of text-spatially aligned reasoning content that adheres to human preferences. Besides, MARE introduces a forgery disentanglement module to capture intrinsic forgery traces from high-level facial semantics, thereby improving its authenticity detection capability. We conduct thorough evaluations on the reasoning content generated by MARE. Both quantitative and qualitative experimental results demonstrate that MARE achieves state-of-the-art performance in terms of accuracy and reliability.