Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies

TL;DR

This paper advances model-free learning in dynamic population games, specifically Karma economies, by analyzing deep reinforcement learning methods for agents to learn equilibria without full game knowledge.

Contribution

It introduces a framework for agents to learn Nash equilibria in Karma DPGs using deep Q-networks and fictitious play, without requiring complete game models.

Findings

Deep Q-Networks achieve suboptimality bounds related to replay buffer size and population size.

Agents can empirically converge to near-equilibrium configurations using model-free deep RL methods.

The approach supports practical deployment of Karma economies for fair resource allocation.

Abstract

Dynamic Population Games (DPGs) provide a tractable framework for modeling strategic interactions in large populations of self-interested agents, and have been successfully applied to the design of Karma economies, a class of fair non-monetary resource allocation mechanisms. Despite their appealing theoretical properties, existing computational tools for DPGs assume full knowledge of the game model and operate in a centralized fashion, limiting their applicability in realistic settings where agents have access only to their own private experience. This paper takes a step towards addressing this gap by studying model-free equilibrium learning in Karma DPGs. First, we analyze the setting in which a novel agent joins a Karma DPG already at its Stationary Nash Equilibrium (SNE) and learns a policy via Deep Q-Networks (DQN) without knowledge of the game model. Leveraging recent convergence results for DQN, we establish a suboptimality bound consisting of a DQN approximation error of order $O(1/\sqrt{N_s})$ and a mean field perturbation error of order $O(1/N)$, where $N_s$ is the replay buffer size and $N$ is the population size. Second, we consider the challenging problem of learning the SNE from scratch. We show empirically that combining deep RL with fictitious play and smoothed policy iteration allows agents to converge, in a model-free fashion, to a configuration close to the centrally computed SNE. Together, these contributions support the vision of Karma economies as practical tools for fair resource allocation.