Convex function approximations for Markov decision processes

TL;DR

This paper develops convex function approximation methods for finite horizon Markov decision processes, proving uniform convergence and demonstrating their effectiveness in financial option pricing with fast computation.

Contribution

It introduces new convex approximation techniques with convergence guarantees and applies them to efficiently compute bounds for Bermudan put options.

Findings

Uniform convergence of convex approximations on compact sets.

Convex bounds can be computed rapidly, in fractions of a CPU second.

Approach can be adapted to concave Bellman functions in minimization problems.

Abstract

This paper studies function approximation for finite horizon discrete time Markov decision processes under certain convexity assumptions. Uniform convergence of these approximations on compact sets is proved under several sampling schemes for the driving random variables. Under some conditions, these approximations form a monotone sequence of lower or upper bounding functions. Numerical experiments involving piecewise linear functions demonstrate that very tight bounding functions for the fair price of a Bermudan put option can be obtained with excellent speed (fractions of a cpu second). Results in this paper can be easily adapted to minimization problems involving concave Bellman functions.