Scale-Agnostic Kolmogorov-Arnold Geometry in Neural Networks

TL;DR

This paper demonstrates that neural networks trained on high-dimensional data like MNIST naturally develop scale-invariant Kolmogorov-Arnold geometric structures during training, revealing organized geometric properties across multiple spatial scales.

Contribution

The study extends the analysis of Kolmogorov-Arnold geometry to realistic high-dimensional data, showing its scale-agnostic emergence in neural networks trained on MNIST.

Findings

KAG emerges during training on MNIST

KAG appears consistently across multiple spatial scales

Scale-invariance of KAG holds under different training procedures

Abstract

Recent work by Freedman and Mulligan demonstrated that shallow multilayer perceptrons spontaneously develop Kolmogorov-Arnold geometric (KAG) structure during training on synthetic three-dimensional tasks. However, it remained unclear whether this phenomenon persists in realistic high-dimensional settings and what spatial properties this geometry exhibits. We extend KAG analysis to MNIST digit classification (784 dimensions) using 2-layer MLPs with systematic spatial analysis at multiple scales. We find that KAG emerges during training and appears consistently across spatial scales, from local 7-pixel neighborhoods to the full 28x28 image. This scale-agnostic property holds across different training procedures: both standard training and training with spatial augmentation produce the same qualitative pattern. These findings reveal that neural networks spontaneously develop organized, scale-invariant geometric structure during learning on realistic high-dimensional data.