BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization

TL;DR

BiERL introduces a bilevel optimization framework for meta reinforcement learning that jointly updates hyperparameters during training, improving exploration and performance across diverse ERL algorithms without prior domain knowledge.

Contribution

The paper proposes a novel bilevel optimization-based meta ERL framework that enables parallel hyperparameter tuning within a single agent, enhancing learning efficiency and robustness.

Findings

BiERL outperforms various baselines in MuJoCo and Box2D tasks.

It consistently improves the learning performance of different ERL algorithms.

The framework reduces the need for prior domain knowledge and costly hyperparameter tuning.

Abstract

Evolutionary reinforcement learning (ERL) algorithms recently raise attention in tackling complex reinforcement learning (RL) problems due to high parallelism, while they are prone to insufficient exploration or model collapse without carefully tuning hyperparameters (aka meta-parameters). In the paper, we propose a general meta ERL framework via bilevel optimization (BiERL) to jointly update hyperparameters in parallel to training the ERL model within a single agent, which relieves the need for prior domain knowledge or costly optimization procedure before model deployment. We design an elegant meta-level architecture that embeds the inner-level's evolving experience into an informative population representation and introduce a simple and feasible evaluation of the meta-level fitness function to facilitate learning efficiency. We perform extensive experiments in MuJoCo and Box2D tasks to verify that as a general framework, BiERL outperforms various baselines and consistently improves the learning performance for a diversity of ERL algorithms.