Perceptual MAE for Image Manipulation Localization: A High-level Vision Learner Focusing on Low-level Features

TL;DR

This paper introduces Perceptual MAE, a novel approach that combines high-level semantic understanding with low-level feature analysis to improve image manipulation localization, outperforming existing methods.

Contribution

It reformulates IML as a high-level vision task, enhancing Masked Autoencoder with high-resolution inputs and perceptual loss to unify low- and high-level features.

Findings

Outperforms state-of-the-art methods on five datasets.

Effectively combines low-level and high-level features.

Enhances understanding of object semantics in IML.

Abstract

Nowadays, multimedia forensics faces unprecedented challenges due to the rapid advancement of multimedia generation technology thereby making Image Manipulation Localization (IML) crucial in the pursuit of truth. The key to IML lies in revealing the artifacts or inconsistencies between the tampered and authentic areas, which are evident under pixel-level features. Consequently, existing studies treat IML as a low-level vision task, focusing on allocating tampered masks by crafting pixel-level features such as image RGB noises, edge signals, or high-frequency features. However, in practice, tampering commonly occurs at the object level, and different classes of objects have varying likelihoods of becoming targets of tampering. Therefore, object semantics are also vital in identifying the tampered areas in addition to pixel-level features. This necessitates IML models to carry out a semantic understanding of the entire image. In this paper, we reformulate the IML task as a high-level vision task that greatly benefits from low-level features. Based on such an interpretation, we propose a method to enhance the Masked Autoencoder (MAE) by incorporating high-resolution inputs and a perceptual loss supervision module, which is termed Perceptual MAE (PMAE). While MAE has demonstrated an impressive understanding of object semantics, PMAE can also compensate for low-level semantics with our proposed enhancements. Evidenced by extensive experiments, this paradigm effectively unites the low-level and high-level features of the IML task and outperforms state-of-the-art tampering localization methods on all five publicly available datasets.