Understanding Deep Neural Networks via Linear Separability of Hidden Layers

TL;DR

This paper investigates the linear separability of hidden layers in deep neural networks using Minkowski difference measures, revealing its correlation with training performance and effects of network architecture and activation functions.

Contribution

It introduces Minkowski difference-based measures for evaluating linear separability and links this measure to training success across various network architectures.

Findings

Linear separability correlates with training performance.

Activation functions and network size affect separability.

Numerical experiments validate the measures across multiple architectures.

Abstract

In this paper, we measure the linear separability of hidden layer outputs to study the characteristics of deep neural networks. In particular, we first propose Minkowski difference based linear separability measures (MD-LSMs) to evaluate the linear separability degree of two points sets. Then, we demonstrate that there is a synchronicity between the linear separability degree of hidden layer outputs and the network training performance, i.e., if the updated weights can enhance the linear separability degree of hidden layer outputs, the updated network will achieve a better training performance, and vice versa. Moreover, we study the effect of activation function and network size (including width and depth) on the linear separability of hidden layers. Finally, we conduct the numerical experiments to validate our findings on some popular deep networks including multilayer perceptron (MLP), convolutional neural network (CNN), deep belief network (DBN), ResNet, VGGNet, AlexNet, vision transformer (ViT) and GoogLeNet.