On the Taylor Expansion of Value Functions

TL;DR

This paper introduces a Taylor expansion-based framework for approximate dynamic programming in discrete-time Markov chains, providing bounds on optimality gaps and practical aggregation methods for control problems.

Contribution

It develops a novel second-order Taylor expansion approach to approximate value functions, linking discrete control problems to continuous Brownian control models with analytical performance bounds.

Findings

Bounds on the optimality gap are established for large initial states.

The framework offers a practical aggregation method with performance guarantees.

Controls derived from the Taylor-based approximation perform well in practice.

Abstract

We introduce a framework for approximate dynamic programming that we apply to discrete time chains on $\mathbb{Z}_+^d$ with countable action sets. Our approach is grounded in the approximation of the (controlled) chain's generator by that of another Markov process. In simple terms, our approach stipulates applying a second-order Taylor expansion to the value function to replace the Bellman equation with one in continuous space and time where the transition matrix is reduced to its first and second moments. In some cases, the resulting equation (which we label {\bf TCP}) can be interpreted as corresponding to a Brownian control problem. When tractable, the TCP serves as a useful modeling tool. More generally, the TCP is a starting point for approximation algorithms. We develop bounds on the optimality gap---the sub-optimality introduced by using the control produced by the "Taylored" equation. These bounds can be viewed as a conceptual underpinning, analytical rather than relying on weak convergence arguments, for the good performance of controls derived from Brownian control problems. We prove that, under suitable conditions and for suitably "large" initial states, (i) the optimality gap is smaller than a $1-\alpha$ fraction of the optimal value, where $\alpha\in (0,1)$ is the discount factor, and (ii) the gap can be further expressed as the infinite horizon discounted value with a "lower-order" per period reward. Computationally, our framework leads to an "aggregation" approach with performance guarantees. While the guarantees are grounded in PDE theory, the practical use of this approach requires no knowledge of that theory.