Pausing Policy Learning in Non-stationary Reinforcement Learning

TL;DR

This paper proposes a novel approach to reinforcement learning in non-stationary environments by strategically pausing policy updates, which improves overall performance and reduces regret compared to continuous updating.

Contribution

It introduces a forecasting-based RL framework with an optimal policy hold duration, backed by theoretical analysis and empirical validation across multiple environments.

Findings

Strategic pausing outperforms continuous updates in reward accumulation.

Optimal hold duration sharpens the upper bound on dynamic regret.

Experimental results confirm higher rewards with non-zero hold durations.

Abstract

Real-time inference is a challenge of real-world reinforcement learning due to temporal differences in time-varying environments: the system collects data from the past, updates the decision model in the present, and deploys it in the future. We tackle a common belief that continually updating the decision is optimal to minimize the temporal gap. We propose forecasting an online reinforcement learning framework and show that strategically pausing decision updates yields better overall performance by effectively managing aleatoric uncertainty. Theoretically, we compute an optimal ratio between policy update and hold duration, and show that a non-zero policy hold duration provides a sharper upper bound on the dynamic regret. Our experimental evaluations on three different environments also reveal that a non-zero policy hold duration yields higher rewards compared to continuous decision updates.