Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

TL;DR

This paper introduces a variation-aware reinforcement learning algorithm for Markov decision processes with both endogenous and exogenous uncertainties, providing new regret bounds under dynamic variation constraints.

Contribution

It develops a novel VB-UCRL algorithm that handles exogenous uncertainty and establishes improved regret bounds compared to existing methods.

Findings

Regret bound of at most 0 or S^{1/6}T^{1/12}

Successfully addresses challenges from exogenous uncertainty

Provides theoretical guarantees for RL in uncertain environments

Abstract

Online reinforcement learning (RL) has been widely applied in information processing scenarios, which usually exhibit much uncertainty due to the intrinsic randomness of channels and service demands. In this paper, we consider an un-discounted RL in general Markov decision processes (MDPs) with both endogeneous and exogeneous uncertainty, where both the rewards and state transition probability are unknown to the RL agent and evolve with the time as long as their respective variations do not exceed certain dynamic budget (i.e., upper bound). We first develop a variation-aware Bernstein-based upper confidence reinforcement learning (VB-UCRL), which we allow to restart according to a schedule dependent on the variations. We successfully overcome the challenges due to the exogeneous uncertainty and establish a regret bound of saving at most $\sqrt{S}$ or $S^{\frac{1}{6}}T^{\frac{1}{12}}$ compared with the latest results in the literature, where $S$ denotes the state size of the MDP and $T$ indicates the iteration index of learning steps.