Geometry of Optimization and Implicit Regularization in Deep Learning

TL;DR

This paper explores how the geometry of the parameter space influences implicit regularization and generalization in deep learning, proposing geometry-aware optimization methods to improve model performance.

Contribution

It introduces a geometric perspective on optimization in deep learning and develops an optimization algorithm tailored to this geometry to enhance generalization.

Findings

Generalization is governed by implicit regularization, not network size.

Changing optimization procedures can improve generalization.

Geometry-aware optimization enhances deep learning performance.

Abstract

We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the empirical optimization procedure can improve generalization, even if actual optimization quality is not affected. We do so by studying the geometry of the parameter space of deep networks, and devising an optimization algorithm attuned to this geometry.