VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments

TL;DR

VS-Bench is a new multimodal benchmark designed to evaluate vision language models' strategic abilities in multi-agent environments, addressing a gap in existing single-agent or text-only benchmarks.

Contribution

The paper introduces VS-Bench, a comprehensive benchmark with environments and metrics for assessing VLMs' perception, reasoning, and decision-making in multi-agent scenarios.

Findings

Current VLMs excel at perception but lag in reasoning and decision-making.

The best model achieves 46.6% prediction accuracy and 31.4% normalized return.

Analysis reveals key factors affecting VLMs' strategic performance.

Abstract

Recent advancements in Vision Language Models (VLMs) have expanded their capabilities to interactive agent tasks, yet existing benchmarks remain limited to single-agent or text-only environments. In contrast, real-world scenarios often involve multiple agents interacting within rich visual and textual contexts, posing challenges with both multimodal observations and strategic interactions. To bridge this gap, we introduce Visual Strategic Bench (VS-Bench), a multimodal benchmark that evaluates VLMs for strategic abilities in multi-agent environments. VS-Bench comprises ten vision-grounded environments that cover cooperative, competitive, and mixed-motive interactions. The performance of VLM agents is evaluated across three dimensions: perception measured by element recognition accuracy; strategic reasoning measured by next-action prediction accuracy; and decision-making measured by normalized episode return. Extensive experiments on fifteen leading VLMs show that, although current models exhibit strong perception abilities, there remains a significant gap to optimal performance in reasoning and decision-making, with the best-performing model attaining 46.6% prediction accuracy and 31.4% normalized return. We further analyze the key factors influencing performance, conduct human experiments, and examine failure modes to provide a deeper understanding of VLMs' strategic abilities. By standardizing the evaluation and highlighting the limitations of existing models, we envision VS-Bench as a foundation for future research on strategic multimodal agents. Code and data are available at https://vs-bench.github.io.