Policy Learning with a Language Bottleneck

TL;DR

The paper introduces PLLB, a framework where AI agents generate linguistic rules to improve interpretability and generalization, facilitating better human-AI interaction across diverse tasks.

Contribution

It presents a novel policy learning framework that integrates language models to produce interpretable rules guiding AI behavior, enhancing transparency and human collaboration.

Findings

Agents learn more interpretable behaviors

Rules enable better generalization across tasks

Shared rules improve human-AI coordination

Abstract

Modern AI systems such as self-driving cars and game-playing agents achieve superhuman performance, but often lack human-like generalization, interpretability, and inter-operability with human users. Inspired by the rich interactions between language and decision-making in humans, we introduce Policy Learning with a Language Bottleneck (PLLB), a framework enabling AI agents to generate linguistic rules that capture the high-level strategies underlying rewarding behaviors. PLLB alternates between a *rule generation* step guided by language models, and an *update* step where agents learn new policies guided by rules, even when a rule is insufficient to describe an entire complex policy. Across five diverse tasks, including a two-player signaling game, maze navigation, image reconstruction, and robot grasp planning, we show that PLLB agents are not only able to learn more interpretable and generalizable behaviors, but can also share the learned rules with human users, enabling more effective human-AI coordination. We provide source code for our experiments at https://github.com/meghabyte/bottleneck .