Value iteration for approximate dynamic programming under convexity

TL;DR

This paper introduces a convexity-based value iteration method for infinite horizon Markov decision processes with uncountable state spaces, ensuring convergence and providing bounds, demonstrated on a financial option example.

Contribution

It develops a tractable convexity-based value iteration approach with convergence guarantees for uncountable state spaces, extending dynamic programming methods.

Findings

Convergence of the modified Bellman operators to original fixed points.

Conditions for monotone bounding sequences of fixed points.

Numerical demonstration on a Bermudan put option.

Abstract

This paper studies value iteration for infinite horizon contracting Markov decision processes under convexity assumptions and when the state space is uncountable. The original value iteration is replaced with a more tractable form and the fixed points from the modified Bellman operators will be shown to converge uniformly on compacts sets to their original counterparts. This holds under various sampling approaches for the random disturbances. Moreover, this paper will present conditions in which these fixed points form monotone sequences of lower bounding or upper bounding functions for the original fixed point. This approach is then demonstrated numerically on a perpetual Bermudan put option.