Complexity of Representations in Deep Learning

TL;DR

This paper investigates how deep neural networks transform data representations across layers, analyzing the evolution of data complexity during training and how network design influences class separation.

Contribution

It introduces a complexity measure to analyze the evolution of data complexity in deep networks, providing insights into the effects of architecture and training data.

Findings

Data complexity decreases across layers during training.

Network design impacts the effectiveness of class separation.

Training sample size influences the complexity reduction.

Abstract

Deep neural networks use multiple layers of functions to map an object represented by an input vector progressively to different representations, and with sufficient training, eventually to a single score for each class that is the output of the final decision function. Ideally, in this output space, the objects of different classes achieve maximum separation. Motivated by the need to better understand the inner working of a deep neural network, we analyze the effectiveness of the learned representations in separating the classes from a data complexity perspective. Using a simple complexity measure, a popular benchmarking task, and a well-known architecture design, we show how the data complexity evolves through the network, how it changes during training, and how it is impacted by the network design and the availability of training samples. We discuss the implications of the observations and the potentials for further studies.