Approximate Bilevel Difference Convex Programming for Bayesian Risk Markov Decision Processes

TL;DR

This paper introduces an efficient method for solving Bayesian risk Markov Decision Processes with convex risk measures, producing policies with performance guarantees in uncertain environments.

Contribution

It proposes the approximate bilevel difference convex programming (ABDCP) approach for solving infinite-horizon BR-MDPs, addressing parameter uncertainty more flexibly than traditional methods.

Findings

ABDCP efficiently computes policies offline.

BR-MDP formulation reduces over-conservatism in uncertainty handling.

Empirical results demonstrate improved performance over existing approaches.

Abstract

We consider infinite-horizon Markov Decision Processes where parameters, such as transition probabilities, are unknown and estimated from data. The popular distributionally robust approach to addressing the parameter uncertainty can sometimes be overly conservative. In this paper, we utilize the recently proposed formulation, Bayesian risk Markov Decision Process (BR-MDP), to address parameter (or epistemic) uncertainty in MDPs. To solve the infinite-horizon BR-MDP with a class of convex risk measures, we propose a computationally efficient approach called approximate bilevel difference convex programming (ABDCP). The optimization is performed offline and produces the optimal policy that is represented as a finite state controller with desirable performance guarantees. We also demonstrate the empirical performance of the BR-MDP formulation and the proposed algorithm.