Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff

TL;DR

This paper investigates the tradeoff between low-dimensional feature representations and regularity in deep neural networks, providing theoretical insights into how depth influences learned feature structure and complexity.

Contribution

It introduces finite depth corrections and formalizes the balance between low-dimensionality and regularity, proving the bottleneck structure in features as depth increases.

Findings

Almost all hidden representations become low-dimensional at large depth

Most weight matrices have singular values close to 1, others diminish with depth

Large learning rates are necessary for convergence of deep representations

Abstract

Previous work has shown that DNNs with large depth $L$ and $L_{2}$-regularization are biased towards learning low-dimensional representations of the inputs, which can be interpreted as minimizing a notion of rank $R^{(0)}(f)$ of the learned function $f$, conjectured to be the Bottleneck rank. We compute finite depth corrections to this result, revealing a measure $R^{(1)}$ of regularity which bounds the pseudo-determinant of the Jacobian $\left|Jf(x)\right|_{+}$ and is subadditive under composition and addition. This formalizes a balance between learning low-dimensional representations and minimizing complexity/irregularity in the feature maps, allowing the network to learn the `right' inner dimension. Finally, we prove the conjectured bottleneck structure in the learned features as $L\to\infty$: for large depths, almost all hidden representations are approximately $R^{(0)}(f)$-dimensional, and almost all weight matrices $W_{\ell}$ have $R^{(0)}(f)$ singular values close to 1 while the others are $O(L^{-\frac{1}{2}})$. Interestingly, the use of large learning rates is required to guarantee an order $O(L)$ NTK which in turns guarantees infinite depth convergence of the representations of almost all layers.