ABBEL: LLM Agents Acting through Belief Bottlenecks Expressed in Language

TL;DR

This paper introduces ABBEL, a framework enabling LLM agents to maintain concise, interpretable belief states in language, reducing memory use in multi-step tasks and improving performance through reinforcement learning.

Contribution

The paper proposes a novel belief bottleneck framework for LLM agents, combining language-based belief states with RL training to enhance efficiency and interpretability in sequential decision-making.

Findings

ABBEL maintains near-constant memory over steps.

RL improves belief quality and task performance.

Belief-based agents outperform full context in memory efficiency.

Abstract

As the length of sequential decision-making tasks increases, it becomes computationally impractical to keep full interaction histories in context. We introduce a general framework for LLM agents to maintain concise contexts through multi-step interaction: Acting through Belief Bottlenecks Expressed in Language (ABBEL), and methods to further improve ABBEL agents with RL post-training. ABBEL replaces long multi-step interaction history by a belief state, i.e., a natural language summary of what has been discovered about task-relevant unknowns. Under ABBEL, at each step the agent first updates a prior belief with the most recent observation from the environment to form a posterior belief, then uses only the posterior to select an action. We systematically evaluate frontier models under ABBEL across six diverse multi-step environments, finding that ABBEL supports generating interpretable beliefs while maintaining near-constant memory use over interaction steps. However, bottleneck approaches are generally prone to error propagation, which we observe causing inferior performance when compared to the full context setting due to errors in belief updating. Therefore, we train LLMs to generate and act on beliefs within the ABBEL framework via reinforcement learning (RL). We experiment with belief grading, to reward higher quality beliefs, as well as belief length penalties to reward more compressed beliefs. Our experiments demonstrate the ability of RL to improve ABBEL's performance beyond the full context setting, while using less memory than contemporaneous approaches.