VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents

TL;DR

VisGym introduces a comprehensive suite of 17 environments for evaluating and training vision-language models in complex, multi-step visual tasks, revealing their current limitations and potential pathways for improvement.

Contribution

The paper presents VisGym, a versatile benchmark suite with multi-step solvers for supervised finetuning, enabling systematic evaluation and advancement of multimodal agents in diverse visual interaction tasks.

Findings

Models perform poorly in interactive multi-step tasks, with success rates below 50%.

Long context windows are less effective than truncated histories for models.

Explicit goal signals and demonstrations improve model performance.

Abstract

Modern Vision-Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGym, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: https://visgym.github.io/.