A Local Geometric Interpretation of Feature Extraction in Deep Feedforward Neural Networks

TL;DR

This paper offers a local geometric perspective on how deep neural networks extract low-dimensional features, revealing that optimal weights and features form a low-rank approximation related to the Bayes action, applicable to various layers and activation functions.

Contribution

It introduces a novel local geometric analysis framework for understanding feature extraction in deep feedforward neural networks, applicable to both hidden and output layers with non-vanishing gradients.

Findings

Optimal weights and features form a low-rank approximation.

Analysis applies to hidden and output layers.

Framework validated on supervised learning problems.

Abstract

In this paper, we present a local geometric analysis to interpret how deep feedforward neural networks extract low-dimensional features from high-dimensional data. Our study shows that, in a local geometric region, the optimal weight in one layer of the neural network and the optimal feature generated by the previous layer comprise a low-rank approximation of a matrix that is determined by the Bayes action of this layer. This result holds (i) for analyzing both the output layer and the hidden layers of the neural network, and (ii) for neuron activation functions with non-vanishing gradients. We use two supervised learning problems to illustrate our results: neural network based maximum likelihood classification (i.e., softmax regression) and neural network based minimum mean square estimation. Experimental validation of these theoretical results will be conducted in our future work.