ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding

TL;DR

ASAP introduces a framework that enhances semantic alignment between images and text using large language models and a novel attention mechanism, significantly improving multi-modal media manipulation detection and grounding accuracy.

Contribution

The paper proposes a new semantic alignment learning approach with MGCA, leveraging large language models to improve manipulation detection and grounding in multi-modal media.

Findings

Outperforms existing methods on DGM4 dataset

Enhances focus on manipulated components during training

Achieves significant accuracy improvements

Abstract

We present ASAP, a new framework for detecting and grounding multi-modal media manipulation (DGM4).Upon thorough examination, we observe that accurate fine-grained cross-modal semantic alignment between the image and text is vital for accurately manipulation detection and grounding. While existing DGM4 methods pay rare attention to the cross-modal alignment, hampering the accuracy of manipulation detecting to step further. To remedy this issue, this work targets to advance the semantic alignment learning to promote this task. Particularly, we utilize the off-the-shelf Multimodal Large-Language Models (MLLMs) and Large Language Models (LLMs) to construct paired image-text pairs, especially for the manipulated instances. Subsequently, a cross-modal alignment learning is performed to enhance the semantic alignment. Besides the explicit auxiliary clues, we further design a Manipulation-Guided Cross Attention (MGCA) to provide implicit guidance for augmenting the manipulation perceiving. With the grounding truth available during training, MGCA encourages the model to concentrate more on manipulated components while downplaying normal ones, enhancing the model's ability to capture manipulations. Extensive experiments are conducted on the DGM4 dataset, the results demonstrate that our model can surpass the comparison method with a clear margin.