A Solver + Gradient Descent Training Algorithm for Deep Neural Networks

TL;DR

This paper introduces GDSolver, a hybrid training algorithm for deep neural networks that combines gradient descent with MILP solving to escape local minima, resulting in improved accuracy and efficiency.

Contribution

The paper presents a novel hybrid training algorithm that integrates gradient descent with MILP to enhance deep neural network training performance.

Findings

GDSolver achieves 31.5% lower MSE in less time for regression.

GDSolver attains higher accuracy on MNIST and CIFAR10 with half the training data.

The hybrid method scales well to large datasets and models.

Abstract

We present a novel hybrid algorithm for training Deep Neural Networks that combines the state-of-the-art Gradient Descent (GD) method with a Mixed Integer Linear Programming (MILP) solver, outperforming GD and variants in terms of accuracy, as well as resource and data efficiency for both regression and classification tasks. Our GD+Solver hybrid algorithm, called GDSolver, works as follows: given a DNN $D$ as input, GDSolver invokes GD to partially train $D$ until it gets stuck in a local minima, at which point GDSolver invokes an MILP solver to exhaustively search a region of the loss landscape around the weight assignments of $D$'s final layer parameters with the goal of tunnelling through and escaping the local minima. The process is repeated until desired accuracy is achieved. In our experiments, we find that GDSolver not only scales well to additional data and very large model sizes, but also outperforms all other competing methods in terms of rates of convergence and data efficiency. For regression tasks, GDSolver produced models that, on average, had 31.5% lower MSE in 48% less time, and for classification tasks on MNIST and CIFAR10, GDSolver was able to achieve the highest accuracy over all competing methods, using only 50% of the training data that GD baselines required.