Omega-regular Verification and Control for Distributional Specifications in MDPs

TL;DR

This paper introduces a novel automated method for verifying and controlling Markov decision processes against distributional omega-regular specifications, expanding the scope beyond existing reachability and safety methods.

Contribution

It presents the first automated approach using distributional certificates for omega-regular specifications in MDPs, with sound, complete proof rules and a template-based synthesis algorithm.

Findings

Algorithms run in PSPACE, demonstrating efficiency.

Prototype implementation shows practical applicability.

Method extends verification capabilities to complex distributional specifications.

Abstract

A classical approach to studying Markov decision processes (MDPs) is to view them as state transformers. However, MDPs can also be viewed as distribution transformers, where an MDP under a strategy generates a sequence of probability distributions over MDP states. This view arises in several applications, even as the probabilistic model checking problem becomes much harder compared to the classical state transformer counterpart. It is known that even distributional reachability and safety problems become computationally intractable (Skolem- and positivity-hard). To address this challenge, recent works focused on sound but possibly incomplete methods for verification and control of MDPs under the distributional view. However, existing automated methods are applicable only to distributional reachability, safety and reach-avoidance specifications. In this work, we present the first automated method for verification and control of MDPs with respect to distributional omega-regular specifications. To achieve this, we propose a novel notion of distributional certificates, which are sound and complete proof rules for proving that an MDP under a distributionally memoryless strategy satisfies some distributional omega-regular specification. We then use our distributional certificates to design the first fully automated algorithms for verification and control of MDPs with respect to distributional omega-regular specifications. Our algorithms follow a template-based synthesis approach and provide soundness and relative completeness guarantees, while running in PSPACE. Our prototype implementation demonstrates practical applicability of our algorithms to challenging examples collected from the literature.