IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks

TL;DR

IMPACT is a new distributed reinforcement learning algorithm that improves training speed and stability by combining importance weighting, target networks, and buffer techniques, outperforming existing methods in various environments.

Contribution

IMPACT extends IMPALA with a target network, circular buffer, and truncated importance sampling to enhance training speed and stability in scalable RL.

Findings

Achieves up to 30% reduction in training wall-time compared to IMPALA.

Attains higher rewards in discrete environments while maintaining sample efficiency.

Faster training in continuous control tasks without sacrificing sample efficiency.

Abstract

The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency). In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA), sample efficiency drops significantly. To address this, we propose a new distributed reinforcement learning algorithm, IMPACT. IMPACT extends IMPALA with three changes: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. In discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that of IMPALA. For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.