On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints

TL;DR

This paper addresses the challenge of designing policies in Markov decision processes that ensure reaching a target with minimal discounted cost, providing theoretical insights and practical algorithms for policy synthesis.

Contribution

It establishes the existence of near-optimal policies, characterizes conditions for optimality, proves NP-completeness, and offers exact and approximate algorithms for policy synthesis.

Findings

Optimal policies may not always exist but near-optimal stationary policies do.

Deciding on an optimal stationary deterministic policy is NP-complete.

An exact mixed-integer linear programming algorithm and an efficient LP-based approximation are proposed.

Abstract

We study the synthesis of a policy in a Markov decision process (MDP) following which an agent reaches a target state in the MDP while minimizing its total discounted cost. The problem combines a reachability criterion with a discounted cost criterion and naturally expresses the completion of a task with probabilistic guarantees and optimal transient performance. We first establish that an optimal policy for the considered formulation may not exist but that there always exists a near-optimal stationary policy. We additionally provide a necessary and sufficient condition for the existence of an optimal policy. We then restrict our attention to stationary deterministic policies and show that the decision problem associated with the synthesis of an optimal stationary deterministic policy is NP-complete. Finally, we provide an exact algorithm based on mixed-integer linear programming and propose an efficient approximation algorithm based on linear programming for the synthesis of an optimal stationary deterministic policy.