Optimising Optimisers with Push GP

TL;DR

This paper employs Push GP to automatically evolve novel local and population-based optimisers that generalize well across various continuous problems, outperforming traditional methods like CMA-ES.

Contribution

It introduces a method for automatically designing optimisers using Push GP, demonstrating their generality and effectiveness across multiple continuous optimization tasks.

Findings

Evolved optimisers generalize well to larger and unseen problems.

Some optimisers outperform CMA-ES on certain benchmarks.

Analysis reveals novel strategies not present in conventional optimisers.

Abstract

This work uses Push GP to automatically design both local and population-based optimisers for continuous-valued problems. The optimisers are trained on a single function optimisation landscape, using random transformations to discourage overfitting. They are then tested for generality on larger versions of the same problem, and on other continuous-valued problems. In most cases, the optimisers generalise well to the larger problems. Surprisingly, some of them also generalise very well to previously unseen problems, outperforming existing general purpose optimisers such as CMA-ES. Analysis of the behaviour of the evolved optimisers indicates a range of interesting optimisation strategies that are not found within conventional optimisers, suggesting that this approach could be useful for discovering novel and effective forms of optimisation in an automated manner.