Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space

TL;DR

This paper introduces the neural Hilbert ladder (NHL), a new theoretical framework that characterizes the function space of multi-layer neural networks as an infinite union of RKHSs, providing insights into their learning dynamics and depth separation.

Contribution

It defines the NHL as a generalization of Barron space for multi-layer NNs, establishing theoretical properties, generalization guarantees, and dynamics in the infinite-width limit.

Findings

NHL generalizes Barron space to multiple layers.

Depth separation phenomena are demonstrated in NHLs.

Numerical experiments illustrate feature learning in NHL framework.

Abstract

To characterize the function space explored by neural networks (NNs) is an important aspect of learning theory. In this work, noticing that a multi-layer NN generates implicitly a hierarchy of reproducing kernel Hilbert spaces (RKHSs) - named a neural Hilbert ladder (NHL) - we define the function space as an infinite union of RKHSs, which generalizes the existing Barron space theory of two-layer NNs. We then establish several theoretical properties of the new space. First, we prove a correspondence between functions expressed by L-layer NNs and those belonging to L-level NHLs. Second, we prove generalization guarantees for learning an NHL with a controlled complexity measure. Third, we derive a non-Markovian dynamics of random fields that governs the evolution of the NHL which is induced by the training of multi-layer NNs in an infinite-width mean-field limit. Fourth, we show examples of depth separation in NHLs under the ReLU activation function. Finally, we perform numerical experiments to illustrate the feature learning aspect of NN training through the lens of NHLs.