Emergent Riemannian geometry over learning discrete computations on continuous manifolds

TL;DR

This paper introduces a geometric framework based on Riemannian metrics to understand how neural networks learn discrete computations from continuous data, revealing the geometric signatures of such processes and their impact on generalization.

Contribution

It demonstrates that neural network computations can be understood through Riemannian geometry, showing how discretization and logical operations emerge in the network's representational geometry.

Findings

Discretization and logical operations are reflected in the Riemannian geometry of network representations.

Different learning regimes exhibit contrasting metric and curvature structures.

The geometric framework explains how networks generalize to unseen inputs.

Abstract

Many tasks require mapping continuous input data (e.g. images) to discrete task outputs (e.g. class labels). Yet, how neural networks learn to perform such discrete computations on continuous data manifolds remains poorly understood. Here, we show that signatures of such computations emerge in the representational geometry of neural networks as they learn. By analysing the Riemannian pullback metric across layers of a neural network, we find that network computation can be decomposed into two functions: discretising continuous input features and performing logical operations on these discretised variables. Furthermore, we demonstrate how different learning regimes (rich vs. lazy) have contrasting metric and curvature structures, affecting the ability of the networks to generalise to unseen inputs. Overall, our work provides a geometric framework for understanding how neural networks learn to perform discrete computations on continuous manifolds.