PROMPT-IML: Image Manipulation Localization with Pre-trained Foundation Models Through Prompt Tuning

TL;DR

This paper introduces PROMPT-IML, a novel framework leveraging pre-trained foundation models and feature fusion to improve image manipulation localization, achieving superior performance and robustness across multiple datasets.

Contribution

It is the first to utilize pre-trained visual foundation models specifically for image manipulation localization, enhancing robustness and accuracy.

Findings

Outperforms existing methods on eight fake image datasets.

Demonstrates improved robustness against image post-processing.

Effectively fuses semantic and high-frequency features for tampering detection.

Abstract

Deceptive images can be shared in seconds with social networking services, posing substantial risks. Tampering traces, such as boundary artifacts and high-frequency information, have been significantly emphasized by massive networks in the Image Manipulation Localization (IML) field. However, they are prone to image post-processing operations, which limit the generalization and robustness of existing methods. We present a novel Prompt-IML framework. We observe that humans tend to discern the authenticity of an image based on both semantic and high-frequency information, inspired by which, the proposed framework leverages rich semantic knowledge from pre-trained visual foundation models to assist IML. We are the first to design a framework that utilizes visual foundation models specially for the IML task. Moreover, we design a Feature Alignment and Fusion module to align and fuse features of semantic features with high-frequency features, which aims at locating tampered regions from multiple perspectives. Experimental results demonstrate that our model can achieve better performance on eight typical fake image datasets and outstanding robustness.