When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration

TL;DR

This paper introduces an orthogonal backfill method for compressing latent messages in multi-agent LLM systems, significantly reducing communication costs while maintaining high performance.

Contribution

It proposes a novel orthogonal residual injection technique to mitigate information loss from hard eviction in KV compression, improving multi-agent communication efficiency.

Findings

Achieves 79.8%--89.4% reduction in communication cost.

Maintains performance comparable to full KV relay across nine benchmarks.

Outperforms existing methods on 7 out of 9 benchmarks.

Abstract

Communication in Large Language Model (LLM)-based multi-agent systems is moving beyond discrete tokens to preserve richer context. Recent work such as LatentMAS enables agents to exchange latent messages through full key-value (KV) caches. However, full KV relay incurs high memory and communication cost. We adapt eviction-style KV compression to this setting and introduce Orthogonal Backfill (OBF) to mitigate information loss from hard eviction. OBF injects a low-rank orthogonal residual from discarded KV states into the retained KV states. We evaluate proposed method against full KV relay on nine standard benchmarks spanning mathematical reasoning, coding, and knowledge-intensive QA. It achieves performance comparable to full KV relay while reducing communication cost by 79.8%--89.4%. OBF further improves the performance and achieves the best results on 7 of the 9 benchmarks. This suggests that more information does not necessarily lead to better communication; preserving the most useful information matters more. Our codebase is publicly available on https://github.com/markli404/When-Less-Latent-Leads-to-Better-Relay.