Novelty-based Sample Reuse for Continuous Robotics Control

TL;DR

This paper introduces a novel method called NSR that enhances reinforcement learning in robotics by selectively reusing samples based on state novelty, leading to faster convergence and better success rates without extra time costs.

Contribution

The paper proposes NSR, a new sample reuse strategy that prioritizes rare states for updates, improving learning efficiency in continuous robotics control.

Findings

NSR accelerates convergence in RL algorithms.

NSR improves success rates in robotic tasks.

NSR does not significantly increase computational time.

Abstract

In reinforcement learning, agents collect state information and rewards through environmental interactions, essential for policy refinement. This process is notably time-consuming, especially in complex robotic simulations and real-world applications. Traditional algorithms usually re-engage with the environment after processing a single batch of samples, thereby failing to fully capitalize on historical data. However, frequently observed states, with reliable value estimates, require minimal updates; in contrast, rare observed states necessitate more intensive updates for achieving accurate value estimations. To address uneven sample utilization, we propose Novelty-guided Sample Reuse (NSR). NSR provides extra updates for infrequent, novel states and skips additional updates for frequent states, maximizing sample use before interacting with the environment again. Our experiments show that NSR improves the convergence rate and success rate of algorithms without significantly increasing time consumption. Our code is publicly available at https://github.com/ppksigs/NSR-DDPG-HER.