LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents

TL;DR

This paper demonstrates that current language models lack the ability to maintain private, hidden information during interactive tasks, and proposes a private working memory as a necessary solution for reliable agent behavior.

Contribution

It introduces the concept of Private State Interactive Tasks (PSITs), proves an impossibility theorem for public-only agents, and proposes a private working memory architecture to enable consistent private state maintenance.

Findings

Standard chat-based LLMs fail to maintain hidden secrets across dialogue branches.

Semantic retrieval does not enable true private state maintenance.

A private working memory restores consistency in interactive language agents.

Abstract

As LLMs move from text completion toward autonomous agents, they remain constrained by the standard chat interface, which lacks private working memory. This raises a fundamental question: can agents reliably perform interactive tasks that depend on hidden state? We define Private State Interactive Tasks (PSITs), which require agents to generate and maintain hidden information while producing consistent public responses. We show theoretically that any agent restricted to the public conversation history cannot simultaneously preserve secrecy and consistency in PSITs, yielding an impossibility theorem. To empirically validate this limitation, we introduce a self-consistency testing protocol that evaluates whether agents can maintain a hidden secret across forked dialogue branches. Standard chat-based LLMs and retrieval-based memory baselines fail this test regardless of scale, demonstrating that semantic retrieval does not enable true state maintenance. To address this, we propose a novel architecture incorporating an explicit private working memory; we demonstrate that this mechanism restores consistency, establishing private state as a necessary component for interactive language agents.