Multi-Agent Reinforcement Learning with Temporal Logic Specifications

TL;DR

This paper introduces ALMANAC, a novel multi-agent reinforcement learning method that learns to satisfy multiple temporal logic specifications in unknown, probabilistic environments, with proven correctness and convergence guarantees.

Contribution

It presents the first multi-agent RL framework capable of handling multiple temporal logic specifications with theoretical guarantees.

Findings

Algorithm demonstrates convergence in experiments.

Handles multiple specifications simultaneously.

Applicable to stochastic, unknown environments.

Abstract

In this paper, we study the problem of learning to satisfy temporal logic specifications with a group of agents in an unknown environment, which may exhibit probabilistic behaviour. From a learning perspective these specifications provide a rich formal language with which to capture tasks or objectives, while from a logic and automated verification perspective the introduction of learning capabilities allows for practical applications in large, stochastic, unknown environments. The existing work in this area is, however, limited. Of the frameworks that consider full linear temporal logic or have correctness guarantees, all methods thus far consider only the case of a single temporal logic specification and a single agent. In order to overcome this limitation, we develop the first multi-agent reinforcement learning technique for temporal logic specifications, which is also novel in its ability to handle multiple specifications. We provide correctness and convergence guarantees for our main algorithm - ALMANAC (Automaton/Logic Multi-Agent Natural Actor-Critic) - even when using function approximation. Alongside our theoretical results, we further demonstrate the applicability of our technique via a set of preliminary experiments.