VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use

TL;DR

VipAct introduces a multi-agent framework that significantly improves fine-grained visual perception in vision-language models by integrating specialized agents and tool use for detailed reasoning.

Contribution

The paper presents VipAct, a novel multi-agent system that enhances VLMs' visual perception through collaboration and specialized perceptual tools, addressing limitations in detailed pixel-level analysis.

Findings

Significant performance improvements on diverse visual perception benchmarks.

Multi-agent collaboration enhances detailed reasoning capabilities.

Ablation studies confirm the importance of tool use and planning in perception tasks.

Abstract

While vision-language models (VLMs) have demonstrated remarkable performance across various tasks combining textual and visual information, they continue to struggle with fine-grained visual perception tasks that require detailed pixel-level analysis. Effectively eliciting comprehensive reasoning from VLMs on such intricate visual elements remains an open challenge. In this paper, we present VipAct, an agent framework that enhances VLMs by integrating multi-agent collaboration and vision expert models, enabling more precise visual understanding and comprehensive reasoning. VipAct consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks such as image captioning and vision expert models that provide high-precision perceptual information. This multi-agent approach allows VLMs to better perform fine-grained visual perception tasks by synergizing planning, reasoning, and tool use. We evaluate VipAct on benchmarks featuring a diverse set of visual perception tasks, with experimental results demonstrating significant performance improvements over state-of-the-art baselines across all tasks. Furthermore, comprehensive ablation studies reveal the critical role of multi-agent collaboration in eliciting more detailed System-2 reasoning and highlight the importance of image input for task planning. Additionally, our error analysis identifies patterns of VLMs' inherent limitations in visual perception, providing insights into potential future improvements. VipAct offers a flexible and extensible framework, paving the way for more advanced visual perception systems across various real-world applications.