Neural Nets with a Newton Conjugate Gradient Method on Multiple GPUs

TL;DR

This paper introduces a second-order optimization method using Newton conjugate gradient for large-scale neural networks, reducing hyperparameter tuning and improving training efficiency on multi-GPU systems.

Contribution

A novel second-order optimization approach that efficiently approximates Hessian effects without explicit Hessian computation, suitable for large neural networks.

Findings

Outperforms existing optimizers on various neural network tasks

Efficient parallelization on multi-GPU systems

Reduces hyperparameter tuning complexity

Abstract

Training deep neural networks consumes increasing computational resource shares in many compute centers. Often, a brute force approach to obtain hyperparameter values is employed. Our goal is (1) to enhance this by enabling second-order optimization methods with fewer hyperparameters for large-scale neural networks and (2) to perform a survey of the performance optimizers for specific tasks to suggest users the best one for their problem. We introduce a novel second-order optimization method that requires the effect of the Hessian on a vector only and avoids the huge cost of explicitly setting up the Hessian for large-scale networks. We compare the proposed second-order method with two state-of-the-art optimizers on five representative neural network problems, including regression and very deep networks from computer vision or variational autoencoders. For the largest setup, we efficiently parallelized the optimizers with Horovod and applied it to a 8 GPU NVIDIA P100 (DGX-1) machine.