DistSPECTRL: Distributing Specifications in Multi-Agent Reinforcement Learning Systems

TL;DR

This paper introduces DistSPECTRL, a framework for specifying and learning objectives in multi-agent systems, enabling effective distributed planning with local and global goals through a novel composition and communication protocol.

Contribution

It presents a new specification framework that combines local and global objectives, facilitating decentralized learning and coordination in multi-agent reinforcement learning systems.

Findings

Effective learning of policies with local objectives in a distributed setting

Decentralized communication protocols enforce global objectives

Successful application to complex multi-agent planning problems

Abstract

While notable progress has been made in specifying and learning objectives for general cyber-physical systems, applying these methods to distributed multi-agent systems still pose significant challenges. Among these are the need to (a) craft specification primitives that allow expression and interplay of both local and global objectives, (b) tame explosion in the state and action spaces to enable effective learning, and (c) minimize coordination frequency and the set of engaged participants for global objectives. To address these challenges, we propose a novel specification framework that allows natural composition of local and global objectives used to guide training of a multi-agent system. Our technique enables learning expressive policies that allow agents to operate in a coordination-free manner for local objectives, while using a decentralized communication protocol for enforcing global ones. Experimental results support our claim that sophisticated multi-agent distributed planning problems can be effectively realized using specification-guided learning.