Graph-of-Constraints Model Predictive Control for Reactive Multi-agent Task and Motion Planning

TL;DR

The paper introduces GoC-MPC, a novel MPC-based framework that effectively manages partially ordered multi-agent tasks with dynamic coordination and disturbance recovery, improving success rates and efficiency.

Contribution

It presents a generalized sequence-of-constraints MPC framework supporting dynamic, partially ordered multi-agent task planning without prior training or environment models.

Findings

Higher success rates in multi-agent manipulation tasks

Faster TAMP computation compared to baselines

Shorter overall paths in experiments

Abstract

Sequences of interdependent geometric constraints are central to many multi-agent Task and Motion Planning (TAMP) problems. However, existing methods for handling such constraint sequences struggle with partially ordered tasks and dynamic agent assignments. They typically assume static assignments and cannot adapt when disturbances alter task allocations. To overcome these limitations, we introduce Graph-of-Constraints Model Predictive Control (GoC-MPC), a generalized sequence-of-constraints framework integrated with MPC. GoC-MPC naturally supports partially ordered tasks, dynamic agent coordination, and disturbance recovery. By defining constraints over tracked 3D keypoints, our method robustly solves diverse multi-agent manipulation tasks-coordinating agents and adapting online from visual observations alone, without relying on training data or environment models. Experiments demonstrate that GoC-MPC achieves higher success rates, significantly faster TAMP computation, and shorter overall paths compared to recent baselines, establishing it as an efficient and robust solution for multi-agent manipulation under real-world disturbances. Our supplementary video and code can be found at https://sites.google.com/view/goc-mpc/home .