Toward Ambulatory Vision: Learning Visually-Grounded Active View Selection

TL;DR

This paper introduces a new task and framework for actively selecting informative viewpoints in visual question answering, enabling embodied agents to move and gather better visual information without scene memory.

Contribution

It proposes VG-AVS, a novel task for active view selection using only current visual data, and develops a fine-tuned VLM-based framework that improves question answering in embodied scenarios.

Findings

The framework achieves strong QA performance based on viewpoint selection.

It generalizes well to unseen synthetic and real scenes.

Incorporating VG-AVS improves existing scene-exploration QA systems.

Abstract

Vision Language Models (VLMs) excel at visual question answering (VQA) but remain limited to snapshot vision, reasoning from static images. In contrast, embodied agents require ambulatory vision, actively moving to obtain more informative views. We introduce Visually Grounded Active View Selection (VG-AVS), a task that selects the most informative next viewpoint using only the visual information in the current image, without relying on scene memory or external knowledge. To support this task, we construct a synthetic dataset with automatically generated paired query-target views and question-answer prompts. We also propose a framework that fine-tunes pretrained VLMs through supervised fine-tuning (SFT) followed by RL-based policy optimization. Our approach achieves strong question answering performance based on viewpoint selection and generalizes robustly to unseen synthetic and real scenes. Furthermore, incorporating our learned VG-AVS framework into existing scene-exploration-based EQA systems improves downstream question-answering accuracy.