A note on Linear Bottleneck networks and their Transition to Multilinearity

TL;DR

This paper investigates how neural networks with bottleneck layers transition from linear to multilinear functions of weights, revealing that the degree of multilinearity depends on the number of bottlenecks rather than total depth.

Contribution

It introduces the concept that networks with bottleneck layers learn multilinear functions, extending the understanding of neural network behavior beyond the infinite width regime.

Findings

Bottleneck layers induce multilinearity in neural networks.

The degree of multilinearity equals the number of bottleneck layers.

Transition to linearity fails when the infinite width assumption is violated.

Abstract

Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius $O(1)$ around initialization. A necessary condition for this result is that all layers of the network are wide enough, i.e., all widths tend to infinity. However, the transition to linearity breaks down when this infinite width assumption is violated. In this work we show that linear networks with a bottleneck layer learn bilinear functions of the weights, in a ball of radius $O(1)$ around initialization. In general, for $B-1$ bottleneck layers, the network is a degree $B$ multilinear function of weights. Importantly, the degree only depends on the number of bottlenecks and not the total depth of the network.