Learning in Games with Quantized Payoff Observations

TL;DR

This paper studies how quantized (coarsely discretized) payoff feedback affects the convergence of multi-agent learning algorithms, revealing thresholds where convergence to equilibrium is preserved or lost.

Contribution

It characterizes the conditions under which quantized feedback still allows convergence to Nash equilibria in FTRL algorithms, highlighting a threshold effect based on quantization error.

Findings

Coarser quantization can still ensure convergence if error is below a game-dependent threshold.

Above the threshold, convergence properties are lost, and players may not learn beyond initial states.

Convergence rate remains unchanged below the quantization threshold.

Abstract

This paper investigates the impact of feedback quantization on multi-agent learning. In particular, we analyze the equilibrium convergence properties of the well-known "follow the regularized leader" (FTRL) class of algorithms when players can only observe a quantized (and possibly noisy) version of their payoffs. In this information-constrained setting, we show that coarser quantization triggers a qualitative shift in the convergence behavior of FTRL schemes. Specifically, if the quantization error lies below a threshold value (which depends only on the underlying game and not on the level of uncertainty entering the process or the specific FTRL variant under study), then (i) FTRL is attracted to the game's strict Nash equilibria with arbitrarily high probability; and (ii) the algorithm's asymptotic rate of convergence remains the same as in the non-quantized case. Otherwise, for larger quantization levels, these convergence properties are lost altogether: players may fail to learn anything beyond their initial state, even with full information on their payoff vectors. This is in contrast to the impact of quantization in continuous optimization problems, where the quality of the obtained solution degrades smoothly with the quantization level.