Meta Mirror Descent: Optimiser Learning for Fast Convergence

TL;DR

Meta Mirror Descent (MetaMD) introduces a theoretically grounded meta-learning framework that optimizes Bregman divergences to accelerate convergence and improve generalisation in machine learning models without needing validation data.

Contribution

It formalizes a novel meta-learning objective based on regret bounds within the mirror descent paradigm, supporting convergence guarantees and broad applicability.

Findings

MetaMD accelerates optimization speed across tasks.

Supports convergence and generalisation guarantees.

Demonstrates strong performance on various architectures.

Abstract

Optimisers are an essential component for training machine learning models, and their design influences learning speed and generalisation. Several studies have attempted to learn more effective gradient-descent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing optimiser learning methods are intuitively motivated, without clear theoretical support. We take a different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergence. Within this paradigm, we formalise a novel meta-learning objective of minimising the regret bound of learning. The resulting framework, termed Meta Mirror Descent (MetaMD), learns to accelerate optimisation speed. Unlike many meta-learned optimisers, it also supports convergence and generalisation guarantees and uniquely does so without requiring validation data. We evaluate our framework on a variety of tasks and architectures in terms of convergence rate and generalisation error and demonstrate strong performance.