Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

TL;DR

This paper investigates how a finite bottleneck in infinitely wide neural networks enables data-dependent feature learning, leading to accelerated training and improved performance compared to standard infinite networks.

Contribution

It introduces the concept that a finite bottleneck in infinite networks allows for feature learning and acceleration, contrasting with the neural tangent kernel limit.

Findings

A single bottleneck accelerates training significantly.

Bottleneck networks outperform purely infinite networks.

The acceleration effect is theoretically explained using deep linear models.

Abstract

We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network al-lows data dependent feature learning in its bottle-neck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely in-finite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.