Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning

TL;DR

This paper introduces NVMDPs, a flexible framework for reinforcement learning that handles non-stationarity and varying discount rates, extending traditional MDPs and improving algorithm robustness in dynamic environments.

Contribution

The paper develops the NVMDP framework, extending MDPs to non-stationary, varying-discount settings, and adapts dynamic programming and policy gradient algorithms with convergence guarantees.

Findings

NVMDP algorithms recover optimal trajectories in non-stationary environments.

Traditional Q-learning fails under non-stationarity, while NVMDP-based methods succeed.

Theoretical foundations and convergence proofs for NVMDP algorithms are established.

Abstract

Algorithms developed under stationary Markov Decision Processes (MDPs) often face challenges in non-stationary environments, and infinite-horizon formulations may not directly apply to finite-horizon tasks. To address these limitations, we introduce the Non-stationary and Varying-discounting MDP (NVMDP) framework, which naturally accommodates non-stationarity and allows discount rates to vary with time and transitions. Infinite-horizon, stationary MDPs emerge as special cases of NVMDPs for identifying an optimal policy, and finite-horizon MDPs are also subsumed within the NVMDP formulations. Moreover, NVMDPs provide a flexible mechanism to shape optimal policies, without altering the state space, action space, or the reward structure. We establish the theoretical foundations of NVMDPs, including assumptions, state- and action-value formulation and recursion, matrix representation, optimality conditions, and policy improvement under finite state and action spaces. Building on these results, we adapt dynamic programming and generalized Q-learning algorithms to NVMDPs, along with formal convergence proofs. For problems requiring function approximation, we extend the Policy Gradient Theorem and the policy improvement bound in Trust Region Policy Optimization (TRPO), offering proofs in both scalar and matrix forms. Empirical evaluations in a non-stationary gridworld environment demonstrate that NVMDP-based algorithms successfully recover optimal trajectories under multiple reward and discounting schemes, whereas original Q-learning fails. These results collectively show that NVMDPs provide a theoretically sound and practically effective framework for reinforcement learning, requiring only minor algorithmic modifications while enabling robust handling of non-stationarity and explicit optimal policy shaping.