Optimizing Millions of Hyperparameters by Implicit Differentiation

TL;DR

This paper introduces a gradient-based hyperparameter optimization algorithm that efficiently handles millions of hyperparameters by combining the implicit function theorem with inverse Hessian approximations, enabling practical tuning of complex models.

Contribution

The authors develop a novel method that leverages implicit differentiation and Hessian approximations to optimize vast hyperparameter spaces with minimal additional computational cost.

Findings

Successfully trained models with millions of hyperparameters.

Efficient hyperparameter tuning comparable in cost to standard training.

Demonstrated effectiveness on data-augmentation networks.

Abstract

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.