Reinforcement Learning in MDPs with Information-Ordered Policies

TL;DR

This paper introduces an epoch-based reinforcement learning algorithm for infinite-horizon average-cost MDPs that uses a partial order over policies to enable counterfactual inference, achieving regret bounds independent of state and action space sizes.

Contribution

It presents a novel algorithm leveraging partial orders over policies for efficient reinforcement learning in MDPs, with theoretical guarantees and broad applicability.

Findings

Achieves regret bound of O(√(w log(|Θ|) T)) independent of state/action space sizes.

Applies to inventory control and queuing systems, providing new theoretical guarantees.

Demonstrates strong empirical results without extra assumptions.

Abstract

We propose an epoch-based reinforcement learning algorithm for infinite-horizon average-cost Markov decision processes (MDPs) that leverages a partial order over a policy class. In this structure, $\pi' \leq \pi$ if data collected under $\pi$ can be used to estimate the performance of $\pi'$, enabling counterfactual inference without additional environment interaction. Leveraging this partial order, we show that our algorithm achieves a regret bound of $O(\sqrt{w \log(|\Theta|) T})$, where $w$ is the width of the partial order. Notably, the bound is independent of the state and action space sizes. We illustrate the applicability of these partial orders in many domains in operations research, including inventory control and queuing systems. For each, we apply our framework to that problem, yielding new theoretical guarantees and strong empirical results without imposing extra assumptions such as convexity in the inventory model or specialized arrival-rate structure in the queuing model.