Anytime-Constrained Reinforcement Learning

TL;DR

This paper introduces a new framework for reinforcement learning with anytime constraints, providing algorithms for planning and learning in such settings, along with complexity results and approximation methods.

Contribution

It develops a reduction from anytime-constrained cMDPs to unconstrained MDPs and proposes efficient algorithms with provable guarantees for approximate solutions.

Findings

Optimal deterministic policies exist with augmented cumulative costs.

The reduction is fixed-parameter tractable for tabular cMDPs with logarithmic cost precision.

Computing non-trivial approximately optimal policies is NP-hard in general.

Abstract

We introduce and study constrained Markov Decision Processes (cMDPs) with anytime constraints. An anytime constraint requires the agent to never violate its budget at any point in time, almost surely. Although Markovian policies are no longer sufficient, we show that there exist optimal deterministic policies augmented with cumulative costs. In fact, we present a fixed-parameter tractable reduction from anytime-constrained cMDPs to unconstrained MDPs. Our reduction yields planning and learning algorithms that are time and sample-efficient for tabular cMDPs so long as the precision of the costs is logarithmic in the size of the cMDP. However, we also show that computing non-trivial approximately optimal policies is NP-hard in general. To circumvent this bottleneck, we design provable approximation algorithms that efficiently compute or learn an arbitrarily accurate approximately feasible policy with optimal value so long as the maximum supported cost is bounded by a polynomial in the cMDP or the absolute budget. Given our hardness results, our approximation guarantees are the best possible under worst-case analysis.