Statistical Neurodynamics of Deep Networks: Geometry of Signal Spaces

TL;DR

This paper analyzes the geometric properties of signal manifolds in deep neural networks using statistical neurodynamics, revealing how they embed, deform, and converge, with implications for understanding neural information processing.

Contribution

It provides a geometric and dynamical analysis of signal manifolds in deep networks, highlighting conformal embedding and curvature behavior, and discusses finite size effects.

Findings

Manifold embedding is conformal and locally isotropic.

Scalar curvature converges or diverges slowly.

Distance between signals stabilizes as layers grow large.

Abstract

Statistical neurodynamics studies macroscopic behaviors of randomly connected neural networks. We consider a deep layered feedforward network where input signals are processed layer by layer. The manifold of input signals is embedded in a higher dimensional manifold of the next layer as a curved submanifold, provided the number of neurons is larger than that of inputs. We show geometrical features of the embedded manifold, proving that the manifold enlarges or shrinks locally isotropically so that it is always embedded conformally. We study the curvature of the embedded manifold. The scalar curvature converges to a constant or diverges to infinity slowly. The distance between two signals also changes, converging eventually to a stable fixed value, provided both the number of neurons in a layer and the number of layers tend to infinity. This causes a problem, since when we consider a curve in the input space, it is mapped as a continuous curve of fractal nature, but our theory contradictorily suggests that the curve eventually converges to a discrete set of equally spaced points. In reality, the numbers of neurons and layers are finite and thus, it is expected that the finite size effect causes the discrepancies between our theory and reality. We need to further study the discrepancies to understand their implications on information processing.