The Role of Linear Layers in Nonlinear Interpolating Networks

TL;DR

This paper investigates how adding linear layers to deep ReLU networks influences their implicit bias, revealing that such architectures favor functions constant in certain directions and relate to representation costs.

Contribution

It introduces a framework analyzing the implicit bias of overparameterized deep networks with linear layers, linking architecture to representation cost and function space bias.

Findings

Adding linear layers affects the network's representation cost and bias.

Optimal interpolants are constant outside a low-dimensional subspace.

The study connects network architecture to implicit bias and function complexity.

Abstract

This paper explores the implicit bias of overparameterized neural networks of depth greater than two layers. Our framework considers a family of networks of varying depth that all have the same capacity but different implicitly defined representation costs. The representation cost of a function induced by a neural network architecture is the minimum sum of squared weights needed for the network to represent the function; it reflects the function space bias associated with the architecture. Our results show that adding linear layers to a ReLU network yields a representation cost that reflects a complex interplay between the alignment and sparsity of ReLU units. Specifically, using a neural network to fit training data with minimum representation cost yields an interpolating function that is constant in directions perpendicular to a low-dimensional subspace on which a parsimonious interpolant exists.