STLGame: Signal Temporal Logic Games in Adversarial Multi-Agent Systems

TL;DR

STLGame introduces a framework using signal temporal logic games and fictitious self-play to synthesize robust, safe policies for autonomous systems in adversarial multi-agent environments, ensuring worst-case STL satisfaction.

Contribution

The paper proposes a novel gradient-based method with differentiable STL formulas within a game-theoretic framework to achieve Nash equilibrium policies in continuous multi-agent settings.

Findings

Policies are nearly unexploitable and robust against unseen opponents.

The gradient-based approach outperforms reinforcement learning in best response approximation.

Experimental results on vehicle and drone benchmarks validate the method's effectiveness.

Abstract

We study how to synthesize a robust and safe policy for autonomous systems under signal temporal logic (STL) tasks in adversarial settings against unknown dynamic agents. To ensure the worst-case STL satisfaction, we propose STLGame, a framework that models the multi-agent system as a two-player zero-sum game, where the ego agents try to maximize the STL satisfaction and other agents minimize it. STLGame aims to find a Nash equilibrium policy profile, which is the best case in terms of robustness against unseen opponent policies, by using the fictitious self-play (FSP) framework. FSP iteratively converges to a Nash profile, even in games set in continuous state-action spaces. We propose a gradient-based method with differentiable STL formulas, which is crucial in continuous settings to approximate the best responses at each iteration of FSP. We show this key aspect experimentally by comparing with reinforcement learning-based methods to find the best response. Experiments on two standard dynamical system benchmarks, Ackermann steering vehicles and autonomous drones, demonstrate that our converged policy is almost unexploitable and robust to various unseen opponents' policies. All code and additional experimental results can be found on our project website: https://sites.google.com/view/stlgame