SIDBench: A Python Framework for Reliably Assessing Synthetic Image Detection Methods

TL;DR

SIDBench is a comprehensive Python framework that evaluates synthetic image detection methods across diverse datasets and transformations, aiming to bridge the gap between experimental results and real-world performance.

Contribution

The paper introduces SIDBench, a modular benchmarking framework that integrates multiple SID models and datasets to improve evaluation of synthetic image detection methods.

Findings

Evaluates SID methods on diverse, realistic datasets.

Assesses impact of image transformations like JPEG compression.

Provides a flexible, extendable benchmarking tool.

Abstract

The generative AI technology offers an increasing variety of tools for generating entirely synthetic images that are increasingly indistinguishable from real ones. Unlike methods that alter portions of an image, the creation of completely synthetic images presents a unique challenge and several Synthetic Image Detection (SID) methods have recently appeared to tackle it. Yet, there is often a large gap between experimental results on benchmark datasets and the performance of methods in the wild. To better address the evaluation needs of SID and help close this gap, this paper introduces a benchmarking framework that integrates several state-of-the-art SID models. Our selection of integrated models was based on the utilization of varied input features, and different network architectures, aiming to encompass a broad spectrum of techniques. The framework leverages recent datasets with a diverse set of generative models, high level of photo-realism and resolution, reflecting the rapid improvements in image synthesis technology. Additionally, the framework enables the study of how image transformations, common in assets shared online, such as JPEG compression, affect detection performance. SIDBench is available on https://github.com/mever-team/sidbench and is designed in a modular manner to enable easy inclusion of new datasets and SID models.