See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

TL;DR

This paper explores how providing vision-language models with symbolic scene representations can improve their ability to perform precise actions in interactive environments, emphasizing the importance of accurate symbol extraction.

Contribution

It introduces a framework for grounding VLMs with symbolic representations and evaluates their performance across multiple interactive environments, highlighting the impact of symbol accuracy.

Findings

Symbolic information improves VLM performance when accurate.

Performance drops when symbol extraction is unreliable.

Perception quality is a key bottleneck for VLMs in gameplay.

Abstract

Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.