A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs

TL;DR

This paper develops a self-consistent Gaussian Process theory that captures feature learning effects in finite deep neural networks, bridging the gap between infinite-width limits and practical finite models.

Contribution

It introduces a novel theoretical framework that models feature learning in finite DNNs, validated on CNNs and fully connected networks, revealing a transition between learning regimes.

Findings

Good agreement with experiments on a toy CNN model

Identifies a sharp transition between feature learning and lazy learning regimes

Derives finite-DNN effects for non-linear fully connected networks

Abstract

Deep neural networks (DNNs) in the infinite width/channel limit have received much attention recently, as they provide a clear analytical window to deep learning via mappings to Gaussian Processes (GPs). Despite its theoretical appeal, this viewpoint lacks a crucial ingredient of deep learning in finite DNNs, laying at the heart of their success -- feature learning. Here we consider DNNs trained with noisy gradient descent on a large training set and derive a self consistent Gaussian Process theory accounting for strong finite-DNN and feature learning effects. Applying this to a toy model of a two-layer linear convolutional neural network (CNN) shows good agreement with experiments. We further identify, both analytical and numerically, a sharp transition between a feature learning regime and a lazy learning regime in this model. Strong finite-DNN effects are also derived for a non-linear two-layer fully connected network. Our self consistent theory provides a rich and versatile analytical framework for studying feature learning and other non-lazy effects in finite DNNs.