Compositional Planning for Logically Constrained Multi-Agent Markov Decision Processes

TL;DR

This paper introduces a compositional approach using Constrained Markov Decision Processes to synthesize decentralized control policies for multi-agent systems with logical constraints, improving scalability and maintaining high probability guarantees.

Contribution

It presents an assume-guarantee based decomposition method for multi-agent CMDPs, enabling scalable policy synthesis with probabilistic constraint satisfaction.

Findings

Policies achieve near-optimal rewards

Order of magnitude reduction in problem size and execution time

High probability guarantees on constraint satisfaction

Abstract

Designing control policies for large, distributed systems is challenging, especially in the context of critical, temporal logic based specifications (e.g., safety) that must be met with high probability. Compositional methods for such problems are needed for scalability, yet relying on worst-case assumptions for decomposition tends to be overly conservative. In this work, we use the framework of Constrained Markov Decision Processes (CMDPs) to provide an assume-guarantee based decomposition for synthesizing decentralized control policies, subject to logical constraints in a multi-agent setting. The returned policies are guaranteed to satisfy the constraints with high probability and provide a lower bound on the achieved objective reward. We empirically find the returned policies to achieve near-optimal rewards while enjoying an order of magnitude reduction in problem size and execution time.