On the curvature of the loss landscape

TL;DR

This paper explores the geometric properties of the loss landscape in deep learning, specifically scalar curvature, to understand why over-parameterized models generalize well despite finite data.

Contribution

It models the loss landscape as a Riemannian manifold and links its scalar curvature to potential indicators of a model's generalization ability.

Findings

Scalar curvature can be computed analytically for the loss landscape.

Connections between curvature and generalization are identified.

Geometric analysis offers insights into deep learning generalization.

Abstract

One of the main challenges in modern deep learning is to understand why such over-parameterized models perform so well when trained on finite data. A way to analyze this generalization concept is through the properties of the associated loss landscape. In this work, we consider the loss landscape as an embedded Riemannian manifold and show that the differential geometric properties of the manifold can be used when analyzing the generalization abilities of a deep net. In particular, we focus on the scalar curvature, which can be computed analytically for our manifold, and show connections to several settings that potentially imply generalization.