Simultaneously Updating All Persistence Values in Reinforcement Learning

TL;DR

This paper introduces the All-Persistence Bellman Operator, enabling simultaneous updates of multiple persistence values in reinforcement learning, improving efficiency and exploration across different time scales.

Contribution

It proposes a novel operator for reinforcement learning that updates all persistence values at once, extending Q-learning and DQN with proven contraction properties.

Findings

Effective use of transitions at any time scale improves learning.

The approach enhances exploration and performance in Atari games.

The operator maintains contraction, ensuring convergence.

Abstract

In reinforcement learning, the performance of learning agents is highly sensitive to the choice of time discretization. Agents acting at high frequencies have the best control opportunities, along with some drawbacks, such as possible inefficient exploration and vanishing of the action advantages. The repetition of the actions, i.e., action persistence, comes into help, as it allows the agent to visit wider regions of the state space and improve the estimation of the action effects. In this work, we derive a novel All-Persistence Bellman Operator, which allows an effective use of both the low-persistence experience, by decomposition into sub-transition, and the high-persistence experience, thanks to the introduction of a suitable bootstrap procedure. In this way, we employ transitions collected at any time scale to update simultaneously the action values of the considered persistence set. We prove the contraction property of the All-Persistence Bellman Operator and, based on it, we extend classic Q-learning and DQN. After providing a study on the effects of persistence, we experimentally evaluate our approach in both tabular contexts and more challenging frameworks, including some Atari games.