Enabling surrogate-assisted evolutionary reinforcement learning via policy embedding

TL;DR

This paper introduces PE-SAERL, a framework that integrates surrogate models with policy embedding to significantly accelerate evolutionary reinforcement learning, demonstrated by up to 7x faster training on Atari games.

Contribution

It is the first to enable surrogate-assisted ERL through policy embedding, addressing high-dimensional policy challenges and improving training efficiency.

Findings

Training speed up to 7x on Atari games

Outperforms four state-of-the-art algorithms

Effective surrogate integration for high-dimensional policies

Abstract

Evolutionary Reinforcement Learning (ERL) that applying Evolutionary Algorithms (EAs) to optimize the weight parameters of Deep Neural Network (DNN) based policies has been widely regarded as an alternative to traditional reinforcement learning methods. However, the evaluation of the iteratively generated population usually requires a large amount of computational time and can be prohibitively expensive, which may potentially restrict the applicability of ERL. Surrogate is often used to reduce the computational burden of evaluation in EAs. Unfortunately, in ERL, each individual of policy usually represents millions of weights parameters of DNN. This high-dimensional representation of policy has introduced a great challenge to the application of surrogates into ERL to speed up training. This paper proposes a PE-SAERL Framework to at the first time enable surrogate-assisted evolutionary reinforcement learning via policy embedding (PE). Empirical results on 5 Atari games show that the proposed method can perform more efficiently than the four state-of-the-art algorithms. The training process is accelerated up to 7x on tested games, comparing to its counterpart without the surrogate and PE.