Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning

TL;DR

This paper provides a theoretical foundation explaining how neural networks can naturally develop symbolic structures through gradient-based training, using algebraic and geometric principles to connect continuous learning with discrete reasoning.

Contribution

It introduces a measure-space and Wasserstein gradient flow framework to explain the emergence of symbolic structures and algebraic constraints during neural network training.

Findings

Neural parameters evolve via Wasserstein gradient flow under geometric constraints.

Training leads to decoupled optimization trajectories and reduced degrees of freedom.

Symbolic solutions are linked to group invariance and algebraic structures.

Abstract

We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics. By lifting neural parameters to a measure space and modeling training as Wasserstein gradient flow, we show that under geometric constraints, such as group invariance, the parameter measure $\mu_t$ undergoes two concurrent phenomena: (1) a decoupling of the gradient flow into independent optimization trajectories over some potential functions, and (2) a progressive contraction on the degree of freedom. These potentials encode algebraic constraints relevant to the task and act as ring homomorphisms under a commutative semi-ring structure on the measure space. As training progresses, the network transitions from a high-dimensional exploration to compositional representations that comply with algebraic operations and exhibit a lower degree of freedom. We further establish data scaling laws for realizing symbolic tasks, linking representational capacity to the group invariance that facilitates symbolic solutions. This framework charts a principled foundation for understanding and designing neurosymbolic systems that integrate continuous learning with discrete algebraic reasoning.