Capturing Localized Image Artifacts through a CNN-based Hyper-image Representation

TL;DR

This paper introduces a two-stage CNN approach using hyper-image representations to better capture localized artifacts in images, improving performance on small datasets for tasks like image quality estimation and tampering detection.

Contribution

The paper proposes a novel hyper-image representation method with a two-stage CNN to enhance localized artifact detection on small datasets, addressing limitations of existing patch-based approaches.

Findings

Improved accuracy over baseline methods in image quality estimation.

Enhanced detection of image tampering with the proposed hyper-image approach.

Effective on synthetic and real-world vision tasks.

Abstract

Training deep CNNs to capture localized image artifacts on a relatively small dataset is a challenging task. With enough images at hand, one can hope that a deep CNN characterizes localized artifacts over the entire data and their effect on the output. However, on smaller datasets, such deep CNNs may overfit and shallow ones find it hard to capture local artifacts. Thus some image-based small-data applications first train their framework on a collection of patches (instead of the entire image) to better learn the representation of localized artifacts. Then the output is obtained by averaging the patch-level results. Such an approach ignores the spatial correlation among patches and how various patch locations affect the output. It also fails in cases where few patches mainly contribute to the image label. To combat these scenarios, we develop the notion of hyper-image representations. Our CNN has two stages. The first stage is trained on patches. The second stage utilizes the last layer representation developed in the first stage to form a hyper-image, which is used to train the second stage. We show that this approach is able to develop a better mapping between the image and its output. We analyze additional properties of our approach and show its effectiveness on one synthetic and two real-world vision tasks - no-reference image quality estimation and image tampering detection - by its performance improvement over existing strong baselines.