Assume-Guarantee Reinforcement Learning

TL;DR

This paper introduces a modular reinforcement learning framework using assume-guarantee contracts expressed as regular languages, enabling scalable learning in complex environments by synthesizing local controllers with guarantees on system-wide satisfaction probabilities.

Contribution

It proposes a novel assume-guarantee paradigm for RL, translating contracts into rewards and providing a method to compute lower bounds on system satisfaction probabilities.

Findings

Efficiently synthesizes local controllers in modular environments.

Provides a lower bound on the probability of system satisfaction.

Demonstrates effectiveness on various case studies.

Abstract

We present a modular approach to \emph{reinforcement learning} (RL) in environments consisting of simpler components evolving in parallel. A monolithic view of such modular environments may be prohibitively large to learn, or may require unrealizable communication between the components in the form of a centralized controller. Our proposed approach is based on the assume-guarantee paradigm where the optimal control for the individual components is synthesized in isolation by making \emph{assumptions} about the behaviors of neighboring components, and providing \emph{guarantees} about their own behavior. We express these \emph{assume-guarantee contracts} as regular languages and provide automatic translations to scalar rewards to be used in RL. By combining local probabilities of satisfaction for each component, we provide a lower bound on the probability of satisfaction of the complete system. By solving a Markov game for each component, RL can produce a controller for each component that maximizes this lower bound. The controller utilizes the information it receives through communication, observations, and any knowledge of a coarse model of other agents. We experimentally demonstrate the efficiency of the proposed approach on a variety of case studies.