Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)

TL;DR

This study shows that simple RNN agents can spontaneously develop internal models of their partners' abilities in collaborative tasks, but only when environmental pressures encourage social adaptation.

Contribution

Demonstrates that structured partner modelling can emerge in model-free agents without explicit mechanisms, driven by social pressures in cooperative environments.

Findings

Agents develop internal representations of partners' abilities.

Structured partner models emerge when agents influence partner behavior.

Partner modelling arises spontaneously under specific environmental pressures.

Abstract

Humans are remarkably adept at collaboration, able to infer the strengths and weaknesses of new partners in order to work successfully towards shared goals. To build AI systems with this capability, we must first understand its building blocks: does such flexibility require explicit, dedicated mechanisms for modelling others -- or can it emerge spontaneously from the pressures of open-ended cooperative interaction? To investigate this question, we train simple model-free RNN agents to collaborate with a population of diverse partners. Using the `Overcooked-AI' environment, we collect data from thousands of collaborative teams, and analyse agents' internal hidden states. Despite a lack of additional architectural features, inductive biases, or auxiliary objectives, the agents nevertheless develop structured internal representations of their partners' task abilities, enabling rapid adaptation and generalisation to novel collaborators. We investigated these internal models through probing techniques, and large-scale behavioural analysis. Notably, we find that structured partner modelling emerges when agents can influence partner behaviour by controlling task allocation. Our results show that partner modelling can arise spontaneously in model-free agents -- but only under environmental conditions that impose the right kind of social pressure.