AutoLoss: Learning Discrete Schedules for Alternate Optimization

TL;DR

AutoLoss is a meta-learning framework that automatically learns optimal discrete optimization schedules for various machine learning tasks, improving convergence quality and generalizing across different models and datasets.

Contribution

It introduces a generic, data-driven approach to learn discrete optimization schedules for alternating parameter updates in ML problems.

Findings

AutoLoss improves convergence quality across four diverse ML tasks.

The learned schedules are adaptable and generalize to new tasks and datasets.

AutoLoss outperforms manually designed schedules in multiple scenarios.

Abstract

Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -- it can guide and improve the learning of a new task model with different specifications, or on different datasets.