ReCoDe: Reinforcement Learning-based Dynamic Constraint Design for Multi-Agent Coordination

TL;DR

ReCoDe is a decentralized framework that enhances multi-agent control by learning dynamic constraints through reinforcement learning, improving coordination and safety in complex navigation tasks.

Contribution

It introduces a hybrid approach combining optimization-based controllers with reinforcement learning to dynamically adapt constraints for multi-agent coordination.

Findings

ReCoDe outperforms handcrafted controllers and MARL baselines in navigation tasks.

It effectively constrains agent movements to prevent congestion.

The framework adapts reliance on expert controllers dynamically.

Abstract

Constraint-based optimization is a cornerstone of robotics, enabling the design of controllers that reliably encode task and safety requirements such as collision avoidance or formation adherence. However, handcrafted constraints can fail in multi-agent settings that demand complex coordination. We introduce ReCoDe--Reinforcement-based Constraint Design--a decentralized, hybrid framework that merges the reliability of optimization-based controllers with the adaptability of multi-agent reinforcement learning. Rather than discarding expert controllers, ReCoDe improves them by learning additional, dynamic constraints that capture subtler behaviors, for example, by constraining agent movements to prevent congestion in cluttered scenarios. Through local communication, agents collectively constrain their allowed actions to coordinate more effectively under changing conditions. In this work, we focus on applications of ReCoDe to multi-agent navigation tasks requiring intricate, context-based movements and consensus, where we show that it outperforms purely handcrafted controllers, other hybrid approaches, and standard MARL baselines. We give empirical (real robot) and theoretical evidence that retaining a user-defined controller, even when it is imperfect, is more efficient than learning from scratch, especially because ReCoDe can dynamically change the degree to which it relies on this controller.