Information Relaxation and Dual Formulation of Controlled Markov Diffusions

TL;DR

This paper extends the information relaxation duality framework from Markov decision processes to controlled Markov diffusions, providing theoretical results and practical bounds for complex stochastic control problems.

Contribution

It develops a dual formulation for controlled Markov diffusions, establishing duality results and introducing new penalties for tighter bounds in stochastic control.

Findings

Dual bounds are effective in assessing policy optimality.

New penalties improve the accuracy of bounds with minimal additional computation.

The dual approach performs well in a portfolio choice application.

Abstract

Information relaxation and duality in Markov decision processes have been studied recently by several researchers with the goal to derive dual bounds on the value function. In this paper we extend this dual formulation to controlled Markov diffusions: in a similar way we relax the constraint that the decision should be made based on the current information and impose penalty to punish the access to the information in advance. We establish the weak duality, strong duality and complementary slackness results in a parallel way as those in Markov decision processes. We explore the structure of the optimal penalties and expose the connection between Markov decision processes and controlled Markov diffusions. We demonstrate the use of the dual representation for controlled Markov diffusions in a classic dynamic portfolio choice problem. We evaluate the lower bounds on the expected utility by Monte Carlo simulation under a sub-optimal policy, and we propose a new class of penalties to derive upper bounds with little extra computation. The small gaps between the lower bounds and upper bounds indicate that the available policy is near optimal as well as the effectiveness of our proposed penalty in the dual method.