Enabling Agents to Communicate Entirely in Latent Space

TL;DR

This paper introduces Interlat, a novel paradigm enabling agents to communicate directly through continuous latent states of LLMs, bypassing language constraints for improved collaboration and efficiency.

Contribution

It proposes a new latent space communication method for LLM agents, demonstrating superior performance and efficiency over traditional chain-of-thought prompting and baselines.

Findings

Interlat outperforms fine-tuned chain-of-thought prompting and single-agent baselines.

Latent communication enables more exploratory behavior and better utilization of internal states.

Further compression accelerates inference by up to 24 times while maintaining performance.

Abstract

While natural language is the de facto communication medium for LLM-based agents, it presents a fundamental constraint. The process of downsampling rich, internal latent states into discrete tokens inherently limits the depth and nuance of information that can be transmitted, thereby hindering collaborative problem-solving. Inspired by telepathy, which bypasses symbolic language in communication, we propose Interlat (Inter-agent Latent Space Communication), a paradigm that leverages the continuous last hidden states of an LLM as a representation of its thought for direct communication (termed latent communication). An additional learned compression process further compresses latent communication via latent space reasoning. Experiments demonstrate that Interlat outperforms both fine-tuned chain-of-thought (CoT) prompting and single-agent baselines, even across heterogeneous models, promoting more exploratory behavior and enabling genuine utilization of latent information. Further compression not only substantially accelerates inference by up to 24 times but also maintains competitive performance through an efficient information-preserving mechanism. We position this work as a feasibility study of entirely latent space inter-agent communication, and our results highlight its potential, offering valuable insights for future research. Our code is available at https://github.com/XiaoDu-flying/Interlat.