Discovering Evolution Strategies via Meta-Black-Box Optimization

TL;DR

This paper introduces a meta-learning approach to discover effective evolution strategies for black-box optimization, leading to more flexible and generalizable algorithms that outperform traditional methods on various tasks.

Contribution

It presents a novel meta-learning framework that uses a self-attention-based architecture to automatically discover evolution strategies capable of generalizing across diverse problems.

Findings

Meta-learned strategies outperform traditional neuroevolution baselines.

Discovered strategies generalize to unseen problems and settings.

Explicit heuristic forms of the learned strategies remain highly competitive.

Abstract

Optimizing functions without access to gradients is the remit of black-box methods such as evolution strategies. While highly general, their learning dynamics are often times heuristic and inflexible - exactly the limitations that meta-learning can address. Hence, we propose to discover effective update rules for evolution strategies via meta-learning. Concretely, our approach employs a search strategy parametrized by a self-attention-based architecture, which guarantees the update rule is invariant to the ordering of the candidate solutions. We show that meta-evolving this system on a small set of representative low-dimensional analytic optimization problems is sufficient to discover new evolution strategies capable of generalizing to unseen optimization problems, population sizes and optimization horizons. Furthermore, the same learned evolution strategy can outperform established neuroevolution baselines on supervised and continuous control tasks. As additional contributions, we ablate the individual neural network components of our method; reverse engineer the learned strategy into an explicit heuristic form, which remains highly competitive; and show that it is possible to self-referentially train an evolution strategy from scratch, with the learned update rule used to drive the outer meta-learning loop.