The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

TL;DR

The paper introduces the Vision Wormhole, a novel framework enabling efficient, model-agnostic, text-free communication in multi-agent systems by leveraging visual encodings to transfer reasoning traces across heterogeneous models.

Contribution

It proposes a universal visual codec and a hub-and-spoke topology to facilitate scalable, high-bandwidth inter-agent communication without relying on pair-specific translators.

Findings

Reduces communication overhead and runtime in multi-agent reasoning.

Maintains reasoning accuracy comparable to text-based communication.

Demonstrates effectiveness across diverse model architectures.

Abstract

Multi-Agent Systems (MAS) powered by Large Language Models have unlocked advanced collaborative reasoning, yet they remain shackled by the inefficiency of discrete text communication, which imposes significant runtime overhead and information quantization loss. While latent state transfer offers a high-bandwidth alternative, existing approaches either assume homogeneous sender-receiver architectures or rely on pair-specific learned translators, limiting scalability and modularity across diverse model families with disjoint manifolds. In this work, we propose the Vision Wormhole, a novel framework that repurposes the visual interface of Vision-Language Models (VLMs) to enable model-agnostic, text-free communication. By introducing a Universal Visual Codec, we map heterogeneous reasoning traces into a shared continuous latent space and inject them directly into the receiver's visual pathway, effectively treating the vision encoder as a universal port for inter-agent telepathy. Our framework adopts a hub-and-spoke topology to reduce pairwise alignment complexity from O(N^2) to O(N) and leverages a label-free, teacher-student distillation objective to align the high-speed visual channel with the robust reasoning patterns of the text pathway. Extensive experiments across heterogeneous model families (e.g., Qwen-VL, Gemma) demonstrate that the Vision Wormhole reduces end-to-end wall-clock time in controlled comparisons while maintaining reasoning fidelity comparable to standard text-based MAS. Code is available at https://github.com/xz-liu/heterogeneous-latent-mas