State-Novelty Guided Action Persistence in Deep Reinforcement Learning

TL;DR

This paper introduces a dynamic, state-novelty guided action persistence method in deep reinforcement learning that improves sample efficiency without extra value function training by adaptively balancing exploration and exploitation.

Contribution

It proposes a novel, adaptive action persistence technique based on state novelty, avoiding additional value function training and seamlessly integrating with existing exploration strategies.

Findings

Significantly improves sample efficiency in DMControl tasks

Effectively balances exploration and exploitation through smooth scheduling

Can be integrated with various exploration strategies

Abstract

While a powerful and promising approach, deep reinforcement learning (DRL) still suffers from sample inefficiency, which can be notably improved by resorting to more sophisticated techniques to address the exploration-exploitation dilemma. One such technique relies on action persistence (i.e., repeating an action over multiple steps). However, previous work exploiting action persistence either applies a fixed strategy or learns additional value functions (or policy) for selecting the repetition number. In this paper, we propose a novel method to dynamically adjust the action persistence based on the current exploration status of the state space. In such a way, our method does not require training of additional value functions or policy. Moreover, the use of a smooth scheduling of the repeat probability allows a more effective balance between exploration and exploitation. Furthermore, our method can be seamlessly integrated into various basic exploration strategies to incorporate temporal persistence. Finally, extensive experiments on different DMControl tasks demonstrate that our state-novelty guided action persistence method significantly improves the sample efficiency.