Joint-perturbation simultaneous pseudo-gradient

TL;DR

This paper introduces a joint-perturbation zeroth-order optimization method for efficiently approximating Nash equilibria in large, complex games without gradient access, significantly reducing computational resources needed.

Contribution

A novel joint perturbation technique that requires constant utility evaluations per iteration, improving efficiency in equilibrium computation for many-player games.

Findings

Reduces utility evaluations from linear to constant per iteration.

Achieves faster convergence to approximate Nash equilibria.

Demonstrates effectiveness on auction and other strategic games.

Abstract

We study the problem of computing an approximate Nash equilibrium of a game whose strategy space is continuous without access to gradients of the utility function. Such games arise, for example, when players' strategies are represented by the parameters of a neural network. Lack of access to gradients is common in reinforcement learning settings, where the environment is treated as a black box, as well as equilibrium finding in mechanisms such as auctions, where the mechanism's payoffs are discontinuous in the players' actions. To tackle this problem, we turn to zeroth-order optimization techniques that combine pseudo-gradients with equilibrium-finding dynamics. Specifically, we introduce a new technique that requires a number of utility function evaluations per iteration that is constant rather than linear in the number of players. It achieves this by performing a single joint perturbation on all players' strategies, rather than perturbing each one individually. This yields a dramatic improvement for many-player games, especially when the utility function is expensive to compute in terms of wall time, memory, money, or other resources. We evaluate our approach on various games, including auctions, which have important real-world applications. Our approach yields a significant reduction in the run time required to reach an approximate Nash equilibrium.