The Yokai Learning Environment: Tracking Beliefs Over Space and Time

TL;DR

The Yokai Learning Environment (YLE) is a new multi-agent RL benchmark designed to better evaluate cooperative AI by requiring agents to track beliefs and shared knowledge, revealing limitations of existing methods in generalization and internal modeling.

Contribution

We introduce YLE, a novel benchmark that emphasizes belief tracking and reasoning, addressing limitations of the Hanabi Learning Environment in measuring algorithmic progress.

Findings

Leading ZSC methods perform poorly in YLE compared to HLE.

YLE exposes persistent gaps and calibration issues in current algorithms.

Progress on HLE does not necessarily translate to YLE.

Abstract

The ability to cooperate with unknown partners is a central challenge in cooperative AI and widely studied in the form of zero-shot coordination (ZSC), which evaluates an algorithm by measuring the performance of independently trained agents when paired. The Hanabi Learning Environment (HLE) has become the dominant benchmark for ZSC, but recent work has achieved near-perfect inter-seed cross-play performance, limiting its ability to track algorithmic progress. We introduce the Yokai Learning Environment (YLE) - an open-source multi-agent RL benchmark in which effective collaboration requires building common ground by tracking and updating beliefs over moving cards, reasoning under ambiguous hints, and deciding when to terminate the game based on inferred shared knowledge - features absent in the HLE, where beliefs are tied to hand slots and hints are truthful by rule. We evaluate the leading ZSC methods, including High-Entropy IPPO, Other-Play, and Off-Belief Learning, which achieve near-perfect inter-seed cross-play in the HLE, and show that in the YLE they exhibit persistent SP-XP gaps, degraded early-ending calibration, and weaker belief representations in cross-play, indicating failure to maintain consistent internal models with unseen partners. Methods that perform best in the HLE do not perform best in the YLE, indicating that progress measured on a single benchmark may not generalise. Together, these results establish YLE as a challenging new ZSC benchmark.