On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints
Yagiz Savas, Christos K. Verginis, Michael Hibbard, Ufuk Topcu

TL;DR
This paper addresses the challenge of designing policies in Markov decision processes that ensure reaching a target with minimal discounted cost, providing theoretical insights and practical algorithms for policy synthesis.
Contribution
It establishes the existence of near-optimal policies, characterizes conditions for optimality, proves NP-completeness, and offers exact and approximate algorithms for policy synthesis.
Findings
Optimal policies may not always exist but near-optimal stationary policies do.
Deciding on an optimal stationary deterministic policy is NP-complete.
An exact mixed-integer linear programming algorithm and an efficient LP-based approximation are proposed.
Abstract
We study the synthesis of a policy in a Markov decision process (MDP) following which an agent reaches a target state in the MDP while minimizing its total discounted cost. The problem combines a reachability criterion with a discounted cost criterion and naturally expresses the completion of a task with probabilistic guarantees and optimal transient performance. We first establish that an optimal policy for the considered formulation may not exist but that there always exists a near-optimal stationary policy. We additionally provide a necessary and sufficient condition for the existence of an optimal policy. We then restrict our attention to stationary deterministic policies and show that the decision problem associated with the synthesis of an optimal stationary deterministic policy is NP-complete. Finally, we provide an exact algorithm based on mixed-integer linear programming and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFormal Methods in Verification
