A Two-Stage Subspace Trust Region Approach for Deep Neural Network Training

TL;DR

This paper introduces a two-stage subspace trust region method for training neural networks, which accelerates convergence, avoids saddle points, and reduces parameter tuning, demonstrating strong performance on benchmarks.

Contribution

The paper presents a novel second-order optimization method using a two-stage trust region approach in low-dimensional subspaces for neural network training.

Findings

Fast objective decay observed

Effective saddle point avoidance

Strong benchmark performance

Abstract

In this paper, we develop a novel second-order method for training feed-forward neural nets. At each iteration, we construct a quadratic approximation to the cost function in a low-dimensional subspace. We minimize this approximation inside a trust region through a two-stage procedure: first inside the embedded positive curvature subspace, followed by a gradient descent step. This approach leads to a fast objective function decay, prevents convergence to saddle points, and alleviates the need for manually tuning parameters. We show the good performance of the proposed algorithm on benchmark datasets.