The Proof of Kolmogorov-Arnold May Illuminate Neural Network Learning

TL;DR

This paper revisits the foundational proof by Kolmogorov and Arnold, interpreting it as a form of sparsity in neural network Jacobians that may explain the emergence of complex concepts in deep learning.

Contribution

It offers a new interpretation of classical mathematical proofs as a sparsity pattern in neural network Jacobians, linking theory to the emergence of higher-order concepts.

Findings

Interpretation of the proof as Jacobian sparsity pattern

Hypothesis that sparsity facilitates higher-order concept emergence

Proposes experiments to test the sparsity hypothesis

Abstract

Kolmogorov and Arnold, in answering Hilbert's 13th problem (in the context of continuous functions), laid the foundations for the modern theory of Neural Networks (NNs). Their proof divides the representation of a multivariate function into two steps: The first (non-linear) inter-layer map gives a universal embedding of the data manifold into a single hidden layer whose image is patterned in such a way that a subsequent dynamic can then be defined to solve for the second inter-layer map. I interpret this pattern as "minor concentration" of the almost everywhere defined Jacobians of the interlayer map. Minor concentration amounts to sparsity for higher exterior powers of the Jacobians. We present a conceptual argument for how such sparsity may set the stage for the emergence of successively higher order concepts in today's deep NNs and suggest two classes of experiments to test this hypothesis.