Steady-State Planning in Expected Reward Multichain MDPs

TL;DR

This paper introduces a linear programming approach for steady-state planning in multichain MDPs, enabling the synthesis of policies that satisfy specific long-term behavior constraints with formal guarantees.

Contribution

It proposes a novel linear programming method for steady-state planning in multichain MDPs, providing guarantees for stationary policies under general conditions.

Findings

Linear programming solution for multichain MDPs

Guarantees on stationary policy behavior

Applicable to complex, disconnected systems

Abstract

The planning domain has experienced increased interest in the formal synthesis of decision-making policies. This formal synthesis typically entails finding a policy which satisfies formal specifications in the form of some well-defined logic. While many such logics have been proposed with varying degrees of expressiveness and complexity in their capacity to capture desirable agent behavior, their value is limited when deriving decision-making policies which satisfy certain types of asymptotic behavior in general system models. In particular, we are interested in specifying constraints on the steady-state behavior of an agent, which captures the proportion of time an agent spends in each state as it interacts for an indefinite period of time with its environment. This is sometimes called the average or expected behavior of the agent and the associated planning problem is faced with significant challenges unless strong restrictions are imposed on the underlying model in terms of the connectivity of its graph structure. In this paper, we explore this steady-state planning problem that consists of deriving a decision-making policy for an agent such that constraints on its steady-state behavior are satisfied. A linear programming solution for the general case of multichain Markov Decision Processes (MDPs) is proposed and we prove that optimal solutions to the proposed programs yield stationary policies with rigorous guarantees of behavior.