FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant

TL;DR

This paper introduces FFAA, a multimodal large language model-based system for explainable face forgery analysis, addressing challenges of diverse forgery techniques, environmental factors, and the need for interpretability.

Contribution

The paper proposes a new open-world face forgery analysis benchmark and a multimodal model that offers explainable, robust, and accurate forgery detection results.

Findings

Enhanced accuracy and robustness over previous methods

Provides user-friendly and explainable forgery analysis results

Introduces a new dataset with descriptive annotations for face forgery

Abstract

The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery techniques, varied facial features and complex environmental factors pose significant challenges for face forgery analysis. Existing datasets lack descriptive annotations of these aspects, making it difficult for models to distinguish between real and forged faces using only visual information amid various confounding factors. In addition, existing methods fail to yield user-friendly and explainable results, hindering the understanding of the model's decision-making process. To address these challenges, we introduce a novel Open-World Face Forgery Analysis VQA (OW-FFA-VQA) task and its corresponding benchmark. To tackle this task, we first establish a dataset featuring a diverse collection of real and forged face images with essential descriptions and reliable forgery reasoning. Based on this dataset, we introduce FFAA: Face Forgery Analysis Assistant, consisting of a fine-tuned Multimodal Large Language Model (MLLM) and Multi-answer Intelligent Decision System (MIDS). By integrating hypothetical prompts with MIDS, the impact of fuzzy classification boundaries is effectively mitigated, enhancing model robustness. Extensive experiments demonstrate that our method not only provides user-friendly and explainable results but also significantly boosts accuracy and robustness compared to previous methods.