Primal-dual regression approach for Markov decision processes with general state and action space

TL;DR

This paper introduces a regression-based primal-dual martingale method for finite horizon Markov decision processes with complex state and action spaces, enabling tight approximations of value functions and policies with efficient Monte Carlo procedures.

Contribution

It presents a novel primal-dual regression approach that provides tight bounds and approximations for MDPs with general spaces, avoiding nested simulations.

Findings

Provides tight error bounds with polynomial dependence on time horizon.

Achieves sublinear dependence on state and action space dimensions.

Eliminates the need for nested simulations in Monte Carlo procedures.

Abstract

We develop a regression based primal-dual martingale approach for solving finite time horizon MDPs with general state and action space. As a result, our method allows for the construction of tight upper and lower biased approximations of the value functions, and, provides tight approximations to the optimal policy. In particular, we prove tight error bounds for the estimated duality gap featuring polynomial dependence on the time horizon, and sublinear dependence on the cardinality/dimension of the possibly infinite state and action space.From a computational point of view the proposed method is efficient since, in contrast to usual duality-based methods for optimal control problems in the literature, the Monte Carlo procedures here involved do not require nested simulations.