Implicit Regularization in Deep Learning

TL;DR

This paper explores how implicit regularization from optimization algorithms influences generalization in deep learning, analyzing complexity measures and invariances to better understand neural network success.

Contribution

It demonstrates the role of implicit regularization in deep learning and investigates complexity measures and invariances that explain generalization phenomena.

Findings

Implicit regularization significantly impacts generalization.

Certain complexity measures can predict neural network performance.

Invariances in neural networks relate to specific optimization algorithms.

Abstract

In an attempt to better understand generalization in deep learning, we study several possible explanations. We show that implicit regularization induced by the optimization method is playing a key role in generalization and success of deep learning models. Motivated by this view, we study how different complexity measures can ensure generalization and explain how optimization algorithms can implicitly regularize complexity measures. We empirically investigate the ability of these measures to explain different observed phenomena in deep learning. We further study the invariances in neural networks, suggest complexity measures and optimization algorithms that have similar invariances to those in neural networks and evaluate them on a number of learning tasks.