Optimizer Amalgamation

TL;DR

This paper introduces a novel approach called Optimizer Amalgamation, which combines multiple optimizers into a single learnable optimizer using differentiable mechanisms and variance reduction techniques, demonstrating improved performance over individual optimizers.

Contribution

It proposes a new method to amalgamate multiple optimizers into one using gradient-based learning and variance reduction, advancing the field of learning to optimize.

Findings

Amalgamated optimizer outperforms individual optimizers.

Variance reduction improves amalgamation stability.

The approach surpasses existing learning to optimize baselines.

Abstract

Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners. Many analytical optimizers have been proposed using a variety of theoretical and empirical approaches; however, none can offer a universal advantage over other competitive optimizers. We are thus motivated to study a new problem named Optimizer Amalgamation: how can we best combine a pool of "teacher" optimizers into a single "student" optimizer that can have stronger problem-specific performance? In this paper, we draw inspiration from the field of "learning to optimize" to use a learnable amalgamation target. First, we define three differentiable amalgamation mechanisms to amalgamate a pool of analytical optimizers by gradient descent. Then, in order to reduce variance of the amalgamation process, we also explore methods to stabilize the amalgamation process by perturbing the amalgamation target. Finally, we present experiments showing the superiority of our amalgamated optimizer compared to its amalgamated components and learning to optimize baselines, and the efficacy of our variance reducing perturbations. Our code and pre-trained models are publicly available at http://github.com/VITA-Group/OptimizerAmalgamation.