TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization

TL;DR

TruFor is a versatile forensic framework that detects and localizes a wide range of image forgeries by analyzing high-level and low-level clues, ensuring robustness and reliability in forensic analysis.

Contribution

The paper introduces a transformer-based fusion architecture that combines RGB images with learned noise fingerprints for robust forgery detection and localization.

Findings

Outperforms state-of-the-art methods on multiple datasets

Effectively detects both cheapfakes and deepfakes

Provides a reliability map to assess localization confidence

Abstract

In this paper we present TruFor, a forensic framework that can be applied to a large variety of image manipulation methods, from classic cheapfakes to more recent manipulations based on deep learning. We rely on the extraction of both high-level and low-level traces through a transformer-based fusion architecture that combines the RGB image and a learned noise-sensitive fingerprint. The latter learns to embed the artifacts related to the camera internal and external processing by training only on real data in a self-supervised manner. Forgeries are detected as deviations from the expected regular pattern that characterizes each pristine image. Looking for anomalies makes the approach able to robustly detect a variety of local manipulations, ensuring generalization. In addition to a pixel-level localization map and a whole-image integrity score, our approach outputs a reliability map that highlights areas where localization predictions may be error-prone. This is particularly important in forensic applications in order to reduce false alarms and allow for a large scale analysis. Extensive experiments on several datasets show that our method is able to reliably detect and localize both cheapfakes and deepfakes manipulations outperforming state-of-the-art works. Code is publicly available at https://grip-unina.github.io/TruFor/