PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments

TL;DR

This paper introduces Active Visual Reasoning (AVR), a new task for multimodal large language models that involves actively exploring and reasoning in partially observable, interactive environments, supported by a new benchmark and dataset.

Contribution

It proposes the AVR task, creates the CLEVR-AVR benchmark and AVR-152k dataset, and develops PhysVLM-AVR, a model that advances active visual reasoning in complex environments.

Findings

PhysVLM-AVR achieves state-of-the-art results on multiple benchmarks.

Current embodied MLLMs struggle with active information acquisition.

The dataset enables training models for iterative reasoning and decision-making.

Abstract

Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings, limiting their effectiveness in real-world environments where information is often incomplete due to occlusion or limited field of view. Humans, in contrast, actively explore and interact with their environment-moving, examining, and manipulating objects-to gather information through a closed-loop process integrating perception, reasoning, and action. Inspired by this human capability, we introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to partially observable, interactive environments. AVR necessitates agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive environments designed to assess both reasoning correctness and information-gathering efficiency. We present AVR-152k, a large-scale dataset that offers rich Chain-of-Thought (CoT) annotations detailing iterative reasoning for uncertainty identification, action-conditioned information gain prediction, and information-maximizing action selection, crucial for training agents in a higher-order Markov Decision Process. Building on this, we develop PhysVLM-AVR, an MLLM achieving state-of-the-art performance on CLEVR-AVR, embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath, Geometry30K). Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities.