On the Implicit Bias in Deep-Learning Algorithms

TL;DR

This paper reviews the concept of implicit bias in deep learning algorithms, explaining its role in their ability to generalize despite overparameterization, and discusses key theoretical results and implications.

Contribution

It provides a concise survey of implicit bias in deep learning, summarizing main theoretical findings and their significance for understanding generalization.

Findings

Implicit bias influences generalization in deep learning.

Gradient-based algorithms tend to converge to solutions with specific properties.

Understanding implicit bias can inform the design of better algorithms.

Abstract

Gradient-based deep-learning algorithms exhibit remarkable performance in practice, but it is not well-understood why they are able to generalize despite having more parameters than training examples. It is believed that implicit bias is a key factor in their ability to generalize, and hence it was widely studied in recent years. In this short survey, we explain the notion of implicit bias, review main results and discuss their implications.