In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

TL;DR

This paper investigates the true nature of inductive bias in deep learning, emphasizing the role of implicit regularization over network size, and provides experimental evidence supporting this perspective.

Contribution

It introduces the idea that implicit regularization, rather than network capacity, is the key inductive bias in deep neural networks, supported by experimental demonstrations.

Findings

Implicit regularization influences learning in deep networks.

Network size alone does not fully explain generalization.

Analogy to matrix factorization offers insights into inductive bias.

Abstract

We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.