Specification-Guided Learning of Nash Equilibria with High Social Welfare

TL;DR

This paper introduces a novel reinforcement learning framework that uses high-level specifications to efficiently find Nash equilibrium policies in multi-agent systems, maximizing social welfare.

Contribution

It presents a new specification-guided approach for training Nash equilibrium policies that prioritize social welfare, outperforming existing methods.

Findings

Successfully computes Nash equilibrium policies with high social welfare

Outperforms state-of-the-art baselines in equilibrium quality

Demonstrates effectiveness in challenging control problems

Abstract

Reinforcement learning has been shown to be an effective strategy for automatically training policies for challenging control problems. Focusing on non-cooperative multi-agent systems, we propose a novel reinforcement learning framework for training joint policies that form a Nash equilibrium. In our approach, rather than providing low-level reward functions, the user provides high-level specifications that encode the objective of each agent. Then, guided by the structure of the specifications, our algorithm searches over policies to identify one that provably forms an $\epsilon$-Nash equilibrium (with high probability). Importantly, it prioritizes policies in a way that maximizes social welfare across all agents. Our empirical evaluation demonstrates that our algorithm computes equilibrium policies with high social welfare, whereas state-of-the-art baselines either fail to compute Nash equilibria or compute ones with comparatively lower social welfare.