Pre-training-free Image Manipulation Localization through Non-Mutually Exclusive Contrastive Learning

TL;DR

This paper introduces a novel contrastive learning framework called NCL that effectively localizes manipulated regions in images without pre-training, by handling non-mutually exclusive patches through a dual-branch pivot structure and a pivot-consistent loss.

Contribution

The paper proposes the Non-mutually exclusive Contrastive Learning (NCL) framework that addresses the challenge of non-mutually exclusive patches in image manipulation localization, eliminating the need for pre-training.

Findings

Achieves state-of-the-art results on five benchmarks.

Demonstrates robustness on unseen real-life samples.

Operates effectively without any pre-training.

Abstract

Deep Image Manipulation Localization (IML) models suffer from training data insufficiency and thus heavily rely on pre-training. We argue that contrastive learning is more suitable to tackle the data insufficiency problem for IML. Crafting mutually exclusive positives and negatives is the prerequisite for contrastive learning. However, when adopting contrastive learning in IML, we encounter three categories of image patches: tampered, authentic, and contour patches. Tampered and authentic patches are naturally mutually exclusive, but contour patches containing both tampered and authentic pixels are non-mutually exclusive to them. Simply abnegating these contour patches results in a drastic performance loss since contour patches are decisive to the learning outcomes. Hence, we propose the Non-mutually exclusive Contrastive Learning (NCL) framework to rescue conventional contrastive learning from the above dilemma. In NCL, to cope with the non-mutually exclusivity, we first establish a pivot structure with dual branches to constantly switch the role of contour patches between positives and negatives while training. Then, we devise a pivot-consistent loss to avoid spatial corruption caused by the role-switching process. In this manner, NCL both inherits the self-supervised merits to address the data insufficiency and retains a high manipulation localization accuracy. Extensive experiments verify that our NCL achieves state-of-the-art performance on all five benchmarks without any pre-training and is more robust on unseen real-life samples. The code is available at: https://github.com/Knightzjz/NCL-IML.