Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a data-centric interpretability framework using Sparse Autoencoders and LLM-summarizer methods to analyze training dynamics of large language models in multi-agent reinforcement learning, revealing behaviors and potential issues.

Contribution

It presents Meta-Autointerp, a novel method for grouping SAE features into interpretable hypotheses, and demonstrates its effectiveness in understanding complex training behaviors and improving agent performance.

Findings

90% of SAE Meta-Features are statistically significant

Discovered behaviors include role-playing, language switching, and reward hacking

Augmenting system prompts improved agent scores by +14.2%

Abstract

Large language models (LLMs) are increasingly trained in complex Reinforcement Learning, multi-agent environments, making it difficult to understand how behavior changes over training. Sparse Autoencoders (SAEs) have recently shown to be useful for data-centric interpretability. In this work, we analyze large-scale reinforcement learning training runs from the sophisticated environment of Full-Press Diplomacy by applying pretrained SAEs, alongside LLM-summarizer methods. We introduce Meta-Autointerp, a method for grouping SAE features into interpretable hypotheses about training dynamics. We discover fine-grained behaviors including role-playing patterns, degenerate outputs, language switching, alongside high-level strategic behaviors and environment-specific bugs. Through automated evaluation, we validate that 90% of discovered SAE Meta-Features are significant, and find a surprising reward hacking behavior. However, through two user studies, we find that even subjectively interesting and seemingly helpful SAE features may be worse than useless to humans, along with most LLM generated hypotheses. However, a subset of SAE-derived hypotheses are predictively useful for downstream tasks. We further provide validation by augmenting an untrained agent's system prompt, improving the score by +14.2%. Overall, we show that SAEs and LLM-summarizer provide complementary views into agent behavior, and together our framework forms a practical starting point for future data-centric interpretability work on ensuring trustworthy LLM behavior throughout training.