IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning

TL;DR

IMDPrompter enhances SAM's ability to detect image manipulations by automating prompts and enabling cross-view learning, leading to improved generalization across diverse datasets.

Contribution

The paper introduces IMDPrompter, a novel cross-view prompt learning framework that automates prompt generation and improves SAM's performance in image manipulation detection.

Findings

Effective across five datasets including CASIA and NIST16.

Outperforms existing methods in manipulation detection accuracy.

Enables automated, cross-view image manipulation localization.

Abstract

Using extensive training data from SA-1B, the Segment Anything Model (SAM) has demonstrated exceptional generalization and zero-shot capabilities, attracting widespread attention in areas such as medical image segmentation and remote sensing image segmentation. However, its performance in the field of image manipulation detection remains largely unexplored and unconfirmed. There are two main challenges in applying SAM to image manipulation detection: a) reliance on manual prompts, and b) the difficulty of single-view information in supporting cross-dataset generalization. To address these challenges, we develops a cross-view prompt learning paradigm called IMDPrompter based on SAM. Benefiting from the design of automated prompts, IMDPrompter no longer relies on manual guidance, enabling automated detection and localization. Additionally, we propose components such as Cross-view Feature Perception, Optimal Prompt Selection, and Cross-View Prompt Consistency, which facilitate cross-view perceptual learning and guide SAM to generate accurate masks. Extensive experimental results from five datasets (CASIA, Columbia, Coverage, IMD2020, and NIST16) validate the effectiveness of our proposed method.