A Stochastic Bundle Method for Interpolating Networks

TL;DR

This paper introduces BORAT, a stochastic bundle method for training neural networks that achieves zero empirical loss, offers adaptive learning rates, and demonstrates robustness and competitive performance across datasets.

Contribution

The paper presents BORAT, a novel bundle-based optimization method for neural networks that is robust, adaptive, and theoretically convergent in both convex and non-convex settings.

Findings

BORAT matches state-of-the-art generalization performance.

BORAT is more robust to hyperparameter choices.

Theoretical convergence is established for BORAT.

Abstract

We propose a novel method for training deep neural networks that are capable of interpolation, that is, driving the empirical loss to zero. At each iteration, our method constructs a stochastic approximation of the learning objective. The approximation, known as a bundle, is a pointwise maximum of linear functions. Our bundle contains a constant function that lower bounds the empirical loss. This enables us to compute an automatic adaptive learning rate, thereby providing an accurate solution. In addition, our bundle includes linear approximations computed at the current iterate and other linear estimates of the DNN parameters. The use of these additional approximations makes our method significantly more robust to its hyperparameters. Based on its desirable empirical properties, we term our method Bundle Optimisation for Robust and Accurate Training (BORAT). In order to operationalise BORAT, we design a novel algorithm for optimising the bundle approximation efficiently at each iteration. We establish the theoretical convergence of BORAT in both convex and non-convex settings. Using standard publicly available data sets, we provide a thorough comparison of BORAT to other single hyperparameter optimisation algorithms. Our experiments demonstrate BORAT matches the state-of-the-art generalisation performance for these methods and is the most robust.