GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

TL;DR

GameVerse introduces a new benchmark for evaluating how vision-language models can learn from video-based reflection in video games, demonstrating that combining failure analysis and tutorials enhances model performance.

Contribution

The paper presents a comprehensive benchmark with a novel reflect-and-retry paradigm, a hierarchical taxonomy, and advanced evaluation methods for assessing VLMs' learning from video reflections.

Findings

VLMs benefit from video-based reflection in various settings.

Combining failure trajectories and tutorials improves VLM performance.

The reflect-and-retry paradigm effectively measures VLM learning progress.

Abstract

Human gameplay is a visually grounded interaction loop in which players act, reflect on failures, and watch tutorials to refine strategies. Can Vision-Language Models (VLMs) also learn from video-based reflection? We present GameVerse, a comprehensive video game benchmark that enables a reflective visual interaction loop. Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies. To facilitate systematic and scalable evaluation, we also introduce a cognitive hierarchical taxonomy spanning 15 globally popular games, dual action space for both semantic and GUI control, and milestone evaluation using advanced VLMs to quantify progress. Our experiments show that VLMs benefit from video-based reflection in varied settings, and perform best by combining failure trajectories and expert tutorials-a training-free analogue to reinforcement learning (RL) plus supervised fine-tuning (SFT).Our project page is available at https://gameverse-bench.github.io/ . Our code is available at https://github.com/THUSI-Lab/GameVerse .