Gathering and Exploiting Higher-Order Information when Training Large Structured Models

TL;DR

This paper introduces a method for explicitly computing higher-order derivative projections on parameter subspaces in large models, enabling advanced optimization and regularization techniques that consider long-range interactions.

Contribution

It presents a novel approach for exact higher-order derivative computation on parameter partitions, facilitating improved optimization and hyperparameter tuning in large neural networks.

Findings

Enables computation of higher-order derivatives on parameter subsets at reasonable cost.

Allows for per-subset learning rate adaptation for hyperparameter tuning.

Incorporates long-range layer interactions in optimization, improving training dynamics.

Abstract

When training large models, such as neural networks, the full derivatives of order 2 and beyond are usually inaccessible, due to their computational cost. Therefore, among the second-order optimization methods, it is common to bypass the computation of the Hessian by using first-order information, such as the gradient of the parameters (e.g., quasi-Newton methods) or the activations (e.g., K-FAC). In this paper, we focus on the exact and explicit computation of projections of the Hessian and higher-order derivatives on well-chosen subspaces relevant for optimization. Namely, for a given partition of the set of parameters, we compute tensors that can be seen as "higher-order derivatives according to the partition", at a reasonable cost as long as the number of subsets of the partition remains small. Then, we give some examples of how these tensors can be used. First, we show how to compute a learning rate per subset of parameters, which can be used for hyperparameter tuning. Second, we show how to use these tensors at order 2 to construct an optimization method that uses information contained in the Hessian. Third, we show how to use these tensors at order 3 (information contained in the third derivative of the loss) to regularize this optimization method. The resulting training step has several interesting properties, including: it takes into account long-range interactions between the layers of the trained neural network, which is usually not the case in similar methods (e.g., K-FAC); the trajectory of the optimization is invariant under affine layer-wise reparameterization.