Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models

TL;DR

This paper introduces an evolutionary framework that uses large language models to discover novel reinforcement learning algorithms by searching over executable update rules, leading to competitive performance on benchmarks.

Contribution

It extends REvolve to discover RL algorithms beyond reward functions, excluding standard mechanisms, and incorporates hyperparameter refinement via language models.

Findings

Discovered algorithms perform competitively on Gymnasium benchmarks.

The approach successfully finds nonstandard learning rules.

Hyperparameter refinement improves algorithm performance.

Abstract

Reinforcement learning algorithms are defined by their learning update rules, which are typically hand-designed and fixed. We present an evolutionary framework for discovering reinforcement learning algorithms by searching directly over executable update rules that implement complete training procedures. The approach builds on REvolve, an evolutionary system that uses large language models as generative variation operators, and extends it from reward-function discovery to algorithm discovery. To promote the emergence of nonstandard learning rules, the search excludes canonical mechanisms such as actor--critic structures, temporal-difference losses, and value bootstrapping. Because reinforcement learning algorithms are highly sensitive to internal scalar parameters, we introduce a post-evolution refinement stage in which a large language model proposes feasible hyperparameter ranges for each evolved update rule. Evaluated end-to-end by full training runs on multiple Gymnasium benchmarks, the discovered algorithms achieve competitive performance relative to established baselines, including SAC, PPO, DQN, and A2C.