Narrowing the Focus: Learned Optimizers for Pretrained Models

TL;DR

This paper introduces a specialized learned optimizer that combines multiple update directions to adapt to specific training environments, significantly improving performance on image classification tasks over traditional and general learned optimizers.

Contribution

It proposes a novel layer-specific linear combination approach for learned optimizers, tailored to specific models and datasets, enhancing effectiveness and robustness.

Findings

Outperforms traditional optimizers like Adam on image classification.

Shows strong generalization across different datasets and training durations.

Demonstrates robustness to model initialization.

Abstract

In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand-designed and tuning their hyperparameters is a big part of the training process. Learned optimizers have shown some initial promise, but are generally unsuccessful as a general optimization mechanism applicable to every problem. In this work we explore a different direction: instead of learning general optimizers, we instead specialize them to a specific training environment. We propose a novel optimizer technique that learns a layer-specific linear combination of update directions provided by a set of base optimizers, effectively adapting its strategy to the specific model and dataset. When evaluated on image classification tasks, this specialized optimizer significantly outperforms both traditional off-the-shelf methods such as Adam, as well as existing general learned optimizers. Moreover, it demonstrates robust generalization with respect to model initialization, evaluating on unseen datasets, and training durations beyond its meta-training horizon.