An empirical analysis of the optimization of deep network loss surfaces

TL;DR

This paper empirically investigates how different stochastic gradient descent variants optimize high-dimensional, non-convex loss surfaces of deep neural networks, revealing characteristic behaviors at saddle points.

Contribution

It introduces a visualization method for loss surfaces and analyzes the characteristic descent behaviors of optimization algorithms at saddle points.

Findings

Different algorithms choose distinct descent directions at saddle points.

Optimization algorithms exhibit consistent behaviors across runs.

Loss surface visualization reveals multiple local minima and saddle points.

Abstract

The success of deep neural networks hinges on our ability to accurately and efficiently optimize high-dimensional, non-convex functions. In this paper, we empirically investigate the loss functions of state-of-the-art networks, and how commonly-used stochastic gradient descent variants optimize these loss functions. To do this, we visualize the loss function by projecting them down to low-dimensional spaces chosen based on the convergence points of different optimization algorithms. Our observations suggest that optimization algorithms encounter and choose different descent directions at many saddle points to find different final weights. Based on consistency we observe across re-runs of the same stochastic optimization algorithm, we hypothesize that each optimization algorithm makes characteristic choices at these saddle points.