Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games

TL;DR

This paper introduces a kernel-based maximum entropy inverse reinforcement learning method for mean-field games, enabling the inference of complex reward functions from expert data and demonstrating significant improvements over linear models.

Contribution

It develops a novel RKHS-based IRL framework for mean-field games, providing theoretical guarantees and extending to finite-horizon non-stationary scenarios.

Findings

Kernel-based IRL reduces policy recovery error by over an order of magnitude.

The framework is validated on a mean-field traffic routing game.

The method extends to finite-horizon non-stationary settings with convergence guarantees.

Abstract

We consider the maximum causal entropy inverse reinforcement learning (IRL) problem for infinite-horizon stationary mean-field games (MFG), in which we model the unknown reward function within a reproducing kernel Hilbert space (RKHS). This allows the inference of rich and potentially nonlinear reward structures directly from expert demonstrations, in contrast to most existing approaches for MFGs that typically restrict the reward to a linear combination of a fixed finite set of basis functions and rely on finite-horizon formulations. We introduce a Lagrangian relaxation that enables us to reformulate the problem as an unconstrained log-likelihood maximization and obtain a solution via a gradient ascent algorithm. To establish the theoretical consistency of the algorithm, we prove the smoothness of the log-likelihood objective through the Fr\'echet differentiability of the related soft Bellman operators with respect to the parameters in the RKHS. To illustrate the practical advantages of the RKHS formulation, we validate our framework on a mean-field traffic routing game exhibiting state-dependent preference reversal, where the kernel-based method reduces policy recovery error by over an order of magnitude compared to a linear reward baseline with a comparable parameter count. Furthermore, we extend the framework to the finite-horizon non-stationary setting. We demonstrate that the log-likelihood reformulation is structurally unavailable in this regime and instead develop an alternative gradient descent algorithm on the convex dual via Danskin's theorem, establishing smoothness and convergence guarantees.