Tighter Regret Bounds for Contextual Action-Set Reinforcement Learning

TL;DR

This paper introduces improved regret bounds for episodic reinforcement learning with context-dependent action sets, extending the MVP algorithm and providing both minimax and gap-dependent guarantees.

Contribution

It extends the MVP algorithm to handle context-dependent action sets and derives new minimax, stochastic, and gap-dependent regret bounds with theoretical guarantees.

Findings

Established a minimax regret bound of O( H^3 K \u007c L) for adversarial contexts.

Derived a regret bound of O( H^3 K) for stochastic contexts.

Provided a sample complexity bound of O( H^3 / ^2) for fixed context distributions.

Abstract

We study episodic reinforcement learning with fixed reward and transition functions, but with episode-dependent admissible action sets that are observed at the start of each episode. Performance is measured by cumulative regret against the episode-wise optimal value, $\sum_{k=1}^K [V^{*,M^k} - V^{\pi^k,M^k}]$, where $M^k$ represents the action context in the $k$-th episode. We show that the MVP algorithm naturally extends to this framework and enjoys strong theoretical guarantees. In particular, we establish a minimax regret bound of $\widetilde{O}(\sqrt{SAH^3K\log L})$ for adversarial contexts, where $L$ denotes the number of possible contexts. This result implies a regret bound of $\widetilde{O}(\sqrt{SAH^3K})$ for stochastic contexts. We further translate the stochastic regret guarantee into a sample complexity bound of $\widetilde{O}(SAH^3/\epsilon^2)$ for a fixed context distribution. In addition, we derive a gap-dependent regret bound of \[ \widetilde O\left( \inf_{p\in [0,1)} \left( \frac{1}{\Delta_{\min}^{p}} + pK\Delta_{\min}^{p} \right)\log K \cdot \mathrm{poly}(S,A,H) \right), \] where $\Delta_{\min}^{p}$ is the global $p$-trimmed positive-gap floor over suboptimal $(h,s,a)$ triples. This bound can substantially improve upon the minimax rate when the relevant suboptimality gaps are large.