Stochastic Resetting Accelerates Policy Convergence in Reinforcement Learning

TL;DR

This paper demonstrates that stochastic resetting can significantly accelerate policy convergence in reinforcement learning by improving exploration and value propagation, especially in complex or sparse reward environments.

Contribution

It introduces stochastic resetting as a novel, effective mechanism to enhance reinforcement learning convergence, bridging concepts from statistical mechanics and AI.

Findings

Resetting accelerates policy convergence in grid environments.

Random resetting improves deep RL in sparse reward tasks.

Resetting preserves optimal policies while speeding up learning.

Abstract

Stochastic resetting, where a dynamical process is intermittently returned to a fixed reference state, has emerged as a powerful mechanism for optimizing first-passage properties. Existing theory largely treats static, non-learning processes. Here we ask how stochastic resetting interacts with reinforcement learning, where the underlying dynamics adapt through experience. In tabular grid environments, we find that resetting accelerates policy convergence even when it does not reduce the search time of a purely diffusive agent, indicating a novel mechanism beyond classical first-passage optimization. In a continuous control task with neural-network-based value approximation, we show that random resetting improves deep reinforcement learning when exploration is difficult and rewards are sparse. Unlike temporal discounting, resetting preserves the optimal policy while accelerating convergence by truncating long, uninformative trajectories to enhance value propagation. Our results establish stochastic resetting as a simple, tunable mechanism for accelerating learning, translating a canonical phenomenon of statistical mechanics into an optimization principle for reinforcement learning.