Genetic Drift Regularization: on preventing Actor Injection from breaking Evolution Strategies

TL;DR

This paper introduces Genetic Drift Regularization (GDR), a novel method to prevent actor genome drift in Evolution Strategies, improving convergence and stability when combining ES with RL, especially during actor injection.

Contribution

The paper proposes GDR, a regularization technique that maintains actor genomes close to ES distribution, enhancing hybrid ES-RL training stability and performance.

Findings

GDR improves ES convergence on RL-friendly problems.

GDR stabilizes actor injection, preventing performance collapse.

GDR benefits RL training even without ES injection.

Abstract

Evolutionary Algorithms (EA) have been successfully used for the optimization of neural networks for policy search, but they still remain sample inefficient and underperforming in some cases compared to gradient-based reinforcement learning (RL). Various methods combine the two approaches, many of them training a RL algorithm on data from EA evaluations and injecting the RL actor into the EA population. However, when using Evolution Strategies (ES) as the EA, the RL actor can drift genetically far from the the ES distribution and injection can cause a collapse of the ES performance. Here, we highlight the phenomenon of genetic drift where the actor genome and the ES population distribution progressively drift apart, leading to injection having a negative impact on the ES. We introduce Genetic Drift Regularization (GDR), a simple regularization method in the actor training loss that prevents the actor genome from drifting away from the ES. We show that GDR can improve ES convergence on problems where RL learns well, but also helps RL training on other tasks, , fixes the injection issues better than previous controlled injection methods.