Distinguishing Natural and Computer-Generated Images using Multi-Colorspace fused EfficientNet

TL;DR

This paper introduces a multi-colorspace fused EfficientNet model for three-class classification of natural, computer graphics, and GAN images, addressing a real-world forensics challenge with improved accuracy and interpretability.

Contribution

It proposes a novel multi-colorspace fusion approach using EfficientNet for comprehensive image classification in forensic analysis, considering all image generation categories.

Findings

Model outperforms baselines in accuracy and robustness

Humans struggle to distinguish computer-generated images

Model's explanations align with human reasoning

Abstract

The problem of distinguishing natural images from photo-realistic computer-generated ones either addresses natural images versus computer graphics or natural images versus GAN images, at a time. But in a real-world image forensic scenario, it is highly essential to consider all categories of image generation, since in most cases image generation is unknown. We, for the first time, to our best knowledge, approach the problem of distinguishing natural images from photo-realistic computer-generated images as a three-class classification task classifying natural, computer graphics, and GAN images. For the task, we propose a Multi-Colorspace fused EfficientNet model by parallelly fusing three EfficientNet networks that follow transfer learning methodology where each network operates in different colorspaces, RGB, LCH, and HSV, chosen after analyzing the efficacy of various colorspace transformations in this image forensics problem. Our model outperforms the baselines in terms of accuracy, robustness towards post-processing, and generalizability towards other datasets. We conduct psychophysics experiments to understand how accurately humans can distinguish natural, computer graphics, and GAN images where we could observe that humans find difficulty in classifying these images, particularly the computer-generated images, indicating the necessity of computational algorithms for the task. We also analyze the behavior of our model through visual explanations to understand salient regions that contribute to the model's decision making and compare with manual explanations provided by human participants in the form of region markings, where we could observe similarities in both the explanations indicating the powerful nature of our model to take the decisions meaningfully.