Safe Multi-agent Learning via Trapping Regions

TL;DR

This paper introduces a method to ensure convergence and safety in multi-agent learning by using trapping regions, verified through algorithms, with applications in GANs, traffic control, and economic models.

Contribution

It applies the concept of trapping regions from dynamical systems to create safety sets in multi-agent learning, with algorithms for verification and heuristic sampling.

Findings

Verified trapping regions in systems with known dynamics.

Demonstrated safety sets in traffic and economic models.

Improved convergence guarantees in multi-agent learning.

Abstract

One of the main challenges of multi-agent learning lies in establishing convergence of the algorithms, as, in general, a collection of individual, self-serving agents is not guaranteed to converge with their joint policy, when learning concurrently. This is in stark contrast to most single-agent environments, and sets a prohibitive barrier for deployment in practical applications, as it induces uncertainty in long term behavior of the system. In this work, we apply the concept of trapping regions, known from qualitative theory of dynamical systems, to create safety sets in the joint strategy space for decentralized learning. We propose a binary partitioning algorithm for verification that candidate sets form trapping regions in systems with known learning dynamics, and a heuristic sampling algorithm for scenarios where learning dynamics are not known. We demonstrate the applications to a regularized version of Dirac Generative Adversarial Network, a four-intersection traffic control scenario run in a state of the art open-source microscopic traffic simulator SUMO, and a mathematical model of economic competition.