IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization

TL;DR

IMDL-BenCo introduces the first comprehensive benchmark and modular codebase for Image Manipulation Detection & Localization, enabling standardized evaluation, comparison, and analysis of models in this field.

Contribution

It provides a standardized, reusable framework and implements state-of-the-art models, facilitating rigorous evaluation and advancing IMDL research.

Findings

Enhanced model evaluation consistency

Insights into dataset characteristics and model architecture

Benchmarking of 8 state-of-the-art IMDL models

Abstract

A comprehensive benchmark is yet to be established in the Image Manipulation Detection & Localization (IMDL) field. The absence of such a benchmark leads to insufficient and misleading model evaluations, severely undermining the development of this field. However, the scarcity of open-sourced baseline models and inconsistent training and evaluation protocols make conducting rigorous experiments and faithful comparisons among IMDL models challenging. To address these challenges, we introduce IMDL-BenCo, the first comprehensive IMDL benchmark and modular codebase. IMDL-BenCo: i) decomposes the IMDL framework into standardized, reusable components and revises the model construction pipeline, improving coding efficiency and customization flexibility; ii) fully implements or incorporates training code for state-of-the-art models to establish a comprehensive IMDL benchmark; and iii) conducts deep analysis based on the established benchmark and codebase, offering new insights into IMDL model architecture, dataset characteristics, and evaluation standards. Specifically, IMDL-BenCo includes common processing algorithms, 8 state-of-the-art IMDL models (1 of which are reproduced from scratch), 2 sets of standard training and evaluation protocols, 15 GPU-accelerated evaluation metrics, and 3 kinds of robustness evaluation. This benchmark and codebase represent a significant leap forward in calibrating the current progress in the IMDL field and inspiring future breakthroughs. Code is available at: https://github.com/scu-zjz/IMDLBenCo.