Distributed Planning in Hierarchical Factored MDPs

TL;DR

This paper introduces a distributed planning algorithm for hierarchical multiagent systems that efficiently coordinates agents through message passing, enabling scalable and consistent decision-making in complex dynamical environments.

Contribution

It presents a novel hierarchical, distributed planning method that leverages local computations and message passing to handle large-scale multiagent systems efficiently.

Findings

The algorithm achieves globally consistent plans through local message passing.

Reusing plans and messages speeds up computation in structurally similar hierarchies.

The approach scales to exponentially larger problems by decomposing into small subsystems.

Abstract

We present a principled and efficient planning algorithm for collaborative multiagent dynamical systems. All computation, during both the planning and the execution phases, is distributed among the agents; each agent only needs to model and plan for a small part of the system. Each of these local subsystems is small, but once they are combined they can represent an exponentially larger problem. The subsystems are connected through a subsystem hierarchy. Coordination and communication between the agents is not imposed, but derived directly from the structure of this hierarchy. A globally consistent plan is achieved by a message passing algorithm, where messages correspond to natural local reward functions and are computed by local linear programs; another message passing algorithm allows us to execute the resulting policy. When two portions of the hierarchy share the same structure, our algorithm can reuse plans and messages to speed up computation.