Computing the Exact Pareto Front in Average-Cost Multi-Objective Markov Decision Processes

TL;DR

This paper characterizes the exact Pareto front in average-cost multi-objective Markov decision processes, revealing its geometric structure and enabling solutions without explicit MDP solving.

Contribution

It provides a geometric characterization of the Pareto front in average-cost MOMDPs, showing it as a convex polytope with vertices corresponding to deterministic policies.

Findings

The Pareto front is a continuous, piecewise-linear surface.

Vertices of the front correspond to deterministic policies.

Adjacent vertices differ in exactly one state.

Abstract

Many communication and control problems are cast as multi-objective Markov decision processes (MOMDPs). The complete solution to an MOMDP is the Pareto front. Much of the literature approximates this front via scalarization into single-objective MDPs. Recent work has begun to characterize the full front in discounted or simple bi-objective settings by exploiting its geometry. In this work, we characterize the exact front in average-cost MOMDPs. We show that the front is a continuous, piecewise-linear surface lying on the boundary of a convex polytope. Each vertex corresponds to a deterministic policy, and adjacent vertices differ in exactly one state. Each edge is realized as a convex combination of the policies at its endpoints, with the mixing coefficient given in closed form. We apply these results to a remote state estimation problem, where each vertex on the front corresponds to a threshold policy. The exact Pareto front and solutions to certain non-convex MDPs can be obtained without explicitly solving any MDP.