Are DNNs fooled by extremely unrecognizable images?

TL;DR

This paper investigates whether deep neural networks can be fooled by extremely unrecognizable images, introducing sparse fooling images (SFIs) that lack natural object features and demonstrating their effectiveness in deceiving DNNs, especially in deeper layers.

Contribution

The study introduces SFIs, a minimal class of fooling images with no natural object features, and proves their existence for various models, revealing new vulnerabilities in DNNs.

Findings

SFIs can fool DNNs in deeper layers

Complex models are more vulnerable to SFI attacks

Max pooling layers contribute to vulnerability

Abstract

Fooling images are a potential threat to deep neural networks (DNNs). These images are not recognizable to humans as natural objects, such as dogs and cats, but are misclassified by DNNs as natural-object classes with high confidence scores. Despite their original design concept, existing fooling images retain some features that are characteristic of the target objects if looked into closely. Hence, DNNs can react to these features. In this paper, we address the question of whether there can be fooling images with no characteristic pattern of natural objects locally or globally. As a minimal case, we introduce single-color images with a few pixels altered, called sparse fooling images (SFIs). We first prove that SFIs always exist under mild conditions for linear and nonlinear models and reveal that complex models are more likely to be vulnerable to SFI attacks. With two SFI generation methods, we demonstrate that in deeper layers, SFIs end up with similar features to those of natural images, and consequently, fool DNNs successfully. Among other layers, we discovered that the max pooling layer causes the vulnerability against SFIs. The defense against SFIs and transferability are also discussed. This study highlights the new vulnerability of DNNs by introducing a novel class of images that distributes extremely far from natural images.