Out of the Black Box: Properties of deep neural networks and their applications

TL;DR

This paper formalizes key properties of deep neural networks in image recognition, revealing how local regions influence classification and introducing Deepception, a method to deceive networks without prior architecture knowledge.

Contribution

It identifies and formalizes properties like local, spatial, activation-inhibition, and cumulative effects in neural networks, and proposes Deepception to exploit these for fooling models.

Findings

Images can be divided into regions with varying response probabilities.

Some locations act as activators or inhibitors for recognition.

Deepception achieves an 88% fooling ratio on VGG-VDD-19.

Abstract

Deep neural networks are powerful machine learning approaches that have exhibited excellent results on many classification tasks. However, they are considered as black boxes and some of their properties remain to be formalized. In the context of image recognition, it is still an arduous task to understand why an image is recognized or not. In this study, we formalize some properties shared by eight state-of-the-art deep neural networks in order to grasp the principles allowing a given deep neural network to classify an image. Our results, tested on these eight networks, show that an image can be sub-divided into several regions (patches) responding at different degrees of probability (local property). With the same patch, some locations in the image can answer two (or three) orders of magnitude higher than other locations (spatial property). Some locations are activators and others inhibitors (activation-inhibition property). The repetition of the same patch can increase (or decrease) the probability of recognition of an object (cumulative property). Furthermore, we propose a new approach called Deepception that exploits these properties to deceive a deep neural network. We obtain for the VGG-VDD-19 neural network a fooling ratio of 88\%. Thanks to our "Psychophysics" approach, no prior knowledge on the networks architectures is required.