Optimal Actor-Critic Policy with Optimized Training Datasets

TL;DR

This paper introduces a novel dataset optimization strategy for actor-critic algorithms that significantly reduces sample requirements, leading to faster convergence and improved data efficiency in reinforcement learning tasks.

Contribution

It proposes a unique dataset optimization method combining best episode selection, a policy-fitness model, and genetic algorithms to enhance sampling efficiency.

Findings

Improves sampling efficiency in actor-critic algorithms

Achieves faster convergence to optimal policies

Outperforms contemporary AC methods on benchmarks

Abstract

Actor-critic (AC) algorithms are known for their efficacy and high performance in solving reinforcement learning problems, but they also suffer from low sampling efficiency. An AC based policy optimization process is iterative and needs to frequently access the agent-environment system to evaluate and update the policy by rolling out the policy, collecting rewards and states (i.e. samples), and learning from them. It ultimately requires a huge number of samples to learn an optimal policy. To improve sampling efficiency, we propose a strategy to optimize the training dataset that contains significantly less samples collected from the AC process. The dataset optimization is made of a best episode only operation, a policy parameter-fitness model, and a genetic algorithm module. The optimal policy network trained by the optimized training dataset exhibits superior performance compared to many contemporary AC algorithms in controlling autonomous dynamical systems. Evaluation on standard benchmarks show that the method improves sampling efficiency, ensures faster convergence to optima, and is more data-efficient than its counterparts.