Efficient Computation of Blackwell Optimal Policies using Rational Functions

TL;DR

This paper introduces efficient algorithms for computing Blackwell optimal policies in Markov Decision Problems by using rational functions, achieving the first strongly polynomial-time algorithms for deterministic cases and subexponential algorithms for general cases.

Contribution

It develops novel symbolic algorithms that replace numerical evaluations with rational function operations, enabling efficient computation of Blackwell optimal policies.

Findings

First strongly polynomial-time algorithms for deterministic MDPs.

First subexponential-time algorithms for general MDPs.

Extended policy iteration algorithms to the Blackwell criterion.

Abstract

Markov Decision Problems (MDPs) provide a foundational framework for modelling sequential decision-making across diverse domains, guided by optimality criteria such as discounted and average rewards. However, these criteria have inherent limitations: discounted optimality may overly prioritise short-term rewards, while average optimality relies on strong structural assumptions. Blackwell optimality addresses these challenges, offering a robust and comprehensive criterion that ensures optimality under both discounted and average reward frameworks. Despite its theoretical appeal, existing algorithms for computing Blackwell Optimal (BO) policies are computationally expensive or hard to implement. In this paper we describe procedures for computing BO policies using an ordering of rational functions in the vicinity of $1$. We adapt state-of-the-art algorithms for deterministic and general MDPs, replacing numerical evaluations with symbolic operations on rational functions to derive bounds independent of bit complexity. For deterministic MDPs, we give the first strongly polynomial-time algorithms for computing BO policies, and for general MDPs we obtain the first subexponential-time algorithm. We further generalise several policy iteration algorithms, extending the best known upper bounds from the discounted to the Blackwell criterion.