Axiomatizing Neural Networks via Pursuit of Subspaces

TL;DR

This paper proposes an axiomatic geometric framework called Pursuit of Subspaces to better understand neural networks' mechanisms, representations, and generalization, aiming to bridge the gap between empirical success and theoretical understanding.

Contribution

It introduces a novel axiomatic approach that provides geometric explanations for neural network behavior, advancing towards a unified theoretical foundation.

Findings

Provides geometric explanations for neural network representations

Offers insights into architectural mechanisms and generalization

Establishes a set of postulates for understanding deep learning

Abstract

While deep neural networks have achieved remarkable success across a wide range of domains, their underlying mechanisms remain poorly understood, and they are often regarded as black boxes. This gap between empirical performance and theoretical understanding poses a challenge analogous to the pre-axiomatic stage of classical geometry. In this work, we introduce the Pursuit of Subspaces (PoS) hypothesis, an axiomatic framework that formulates neural network behavior through a set of geometric postulates. These axioms, together with their derived consequences, provide a unified perspective on representation, computation, and generalization in both shallow and deep architectures. We show that this framework yields geometric explanations for fundamental questions in deep learning, including representation structure, architectural mechanisms, and generalization behavior, offering a principled step toward a coherent theoretical foundation.