Solution Methods for Constrained Markov Decision Process with Continuous Probability Modulation

TL;DR

This paper introduces solution methods for constrained Markov Decision Processes with continuous probability modulation, enabling efficient handling of large action sets and extending to non-concave rewards, demonstrated on loan delinquency management.

Contribution

It presents a novel approach to approximate continuous action sets by their extreme points and extends optimization techniques to non-concave reward functions using concave envelopes.

Findings

Effective in managing loan delinquencies.

Reduces complexity by replacing continuous actions with extreme points.

Extends solution methods to non-concave reward functions.

Abstract

We propose solution methods for previously-unsolved constrained MDPs in which actions can continuously modify the transition probabilities within some acceptable sets. While many methods have been proposed to solve regular MDPs with large state sets, there are few practical approaches for solving constrained MDPs with large action sets. In particular, we show that the continuous action sets can be replaced by their extreme points when the rewards are linear in the modulation. We also develop a tractable optimization formulation for concave reward functions and, surprisingly, also extend it to non- concave reward functions by using their concave envelopes. We evaluate the effectiveness of the approach on the problem of managing delinquencies in a portfolio of loans.