Weakly Time-Coupled Approximation of Markov Decision Processes

TL;DR

This paper introduces a weakly time-coupled approximation for high-dimensional finite-horizon Markov decision processes, enabling scalable computation of tighter bounds and policies by decoupling cross-stage dependence from the horizon length.

Contribution

The paper develops a novel WTCA method that reduces temporal coupling in MDP approximations, improving scalability and bound tightness compared to existing methods.

Findings

WTCA provides tighter upper bounds than PO and LSM across tested instances.

WTCA's computational complexity is independent of the horizon length.

WTCA achieves near-optimal policies at longer horizons in practical applications.

Abstract

Finite-horizon Markov decision processes (MDPs) with high-dimensional exogenous uncertainty and endogenous states arise in operations and finance, including the valuation and exercise of Bermudan and real options, but face a scalability barrier as computational complexity grows with the horizon. A common approximation represents the value function using basis functions, but methods for fitting weights treat cross-stage optimization differently. Least squares Monte Carlo (LSM) fits weights via backward recursion and regression, avoiding joint optimization but accumulating error over the horizon. Approximate linear programming (ALP) and pathwise optimization (PO) jointly fit weights to produce upper bounds, but temporal coupling causes computational complexity to grow with the horizon. We show this coupling is an artifact of the approximation architecture, and develop a weakly time-coupled approximation (WTCA) where cross-stage dependence is independent of horizon. For any fixed basis function set, the WTCA upper bound is tighter than that of ALP and looser than that of PO, and converges to the optimal policy value as the basis family expands. We extend parallel deterministic block coordinate descent to the stochastic MDP setting exploiting weak temporal coupling. Applied to WTCA, weak coupling yields computational complexity independent of the horizon. Within equal time budget, solving WTCA accommodates more exogenous samples or basis functions than PO, yielding tighter bounds despite PO being tighter for fixed samples and basis functions. On Bermudan option and ethanol production instances, WTCA produces tighter upper bounds than PO and LSM in every instance tested, with near-optimal policies at longer horizons.