Episodic Policy Gradient Training

TL;DR

Episodic Policy Gradient Training (EPGT) introduces a method where episodic memory dynamically optimizes hyperparameters during reinforcement learning, leading to improved policy performance in various environments.

Contribution

The paper presents EPGT, a novel approach that formulates hyperparameter scheduling as a Markov Decision Process and uses episodic memory to adapt hyperparameters during training.

Findings

EPGT improves policy gradient performance across multiple environments.

The method effectively adapts hyperparameters on-the-fly.

Experimental results show enhanced learning efficiency.

Abstract

We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly. Unlike other hyperparameter searches, we formulate hyperparameter scheduling as a standard Markov Decision Process and use episodic memory to store the outcome of used hyperparameters and their training contexts. At any policy update step, the policy learner refers to the stored experiences, and adaptively reconfigures its learning algorithm with the new hyperparameters determined by the memory. This mechanism, dubbed as Episodic Policy Gradient Training (EPGT), enables an episodic learning process, and jointly learns the policy and the learning algorithm's hyperparameters within a single run. Experimental results on both continuous and discrete environments demonstrate the advantage of using the proposed method in boosting the performance of various policy gradient algorithms.