Dual Latent Memory for Visual Multi-agent System

TL;DR

This paper introduces L$^{2}$-VMAS, a dual latent memory framework for visual multi-agent systems that improves scalability and efficiency by decoupling perception and thinking, and employing proactive memory access.

Contribution

It proposes a novel, model-agnostic dual latent memory architecture with entropy-driven triggering to enhance multi-agent collaboration and scalability.

Findings

Achieves 2.7-5.4% accuracy improvement

Reduces token usage by 21.3-44.8%

Effectively breaks the scaling wall in VMAS

Abstract

While Visual Multi-Agent Systems (VMAS) promise to enhance comprehensive abilities through inter-agent collaboration, empirical evidence reveals a counter-intuitive "scaling wall": increasing agent turns often degrades performance while exponentially inflating token costs. We attribute this failure to the information bottleneck inherent in text-centric communication, where converting perceptual and thinking trajectories into discrete natural language inevitably induces semantic loss. To this end, we propose L$^{2}$-VMAS, a novel model-agnostic framework that enables inter-agent collaboration with dual latent memories. Furthermore, we decouple the perception and thinking while dynamically synthesizing dual latent memories. Additionally, we introduce an entropy-driven proactive triggering that replaces passive information transmission with efficient, on-demand memory access. Extensive experiments among backbones, sizes, and multi-agent structures demonstrate that our method effectively breaks the "scaling wall" with superb scalability, improving average accuracy by 2.7-5.4% while reducing token usage by 21.3-44.8%. Codes: https://github.com/YU-deep/L2-VMAS.