Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity

TL;DR

This paper proves a converse to the universal approximation theorem, showing neural network weights encode functions approximating training data, and uses matrix decomposition to reveal the geometric structure of learned representations and their role in memory capacity.

Contribution

It introduces Layer Matrix Decomposition (LMD) to analyze neural networks, linking eigen-decomposition to data encoding, memory, and recent neural network models.

Findings

Neural network weights encode functions approximating training data.

Matrix decomposition reveals the geometric structure of latent spaces.

Memory capacity and expressivity are interconnected in neural networks.

Abstract

We prove the converse of the universal approximation theorem, i.e. a neural network (NN) encoding theorem which shows that for every stably converged NN of continuous activation functions, its weight matrix actually encodes a continuous function that approximates its training dataset to within a finite margin of error over a bounded domain. We further show that using the Eckart-Young theorem for truncated singular value decomposition of the weight matrix for every NN layer, we can illuminate the nature of the latent space manifold of the training dataset encoded and represented by every NN layer, and the geometric nature of the mathematical operations performed by each NN layer. Our results have implications for understanding how NNs break the curse of dimensionality by harnessing memory capacity for expressivity, and that the two are complementary. This Layer Matrix Decomposition (LMD) further suggests a close relationship between eigen-decomposition of NN layers and the latest advances in conceptualizations of Hopfield networks and Transformer NN models.