IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering

TL;DR

IR3D-Bench introduces a novel benchmark for vision-language models to demonstrate scene understanding through active creation and inverse rendering, emphasizing tool use and geometric accuracy over passive recognition.

Contribution

The paper presents IR3D-Bench, a new benchmark for evaluating vision-language models' ability to perform agentic inverse rendering using tools, moving beyond traditional descriptive tasks.

Findings

Current VLMs show limitations in visual precision during inverse rendering.

The benchmark reveals gaps in tool-using capabilities of state-of-the-art models.

Metrics assess geometric accuracy, spatial relations, and plausibility.

Abstract

Vision-language models (VLMs) excel at descriptive tasks, but whether they truly understand scenes from visual observations remains uncertain. We introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding through active creation rather than passive recognition. Grounded in the analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs) with actively using programming and rendering tools to recreate the underlying 3D structure of an input image, achieving agentic inverse rendering through tool use. This "understanding-by-creating" approach probes the tool-using generative capacity of VLAs, moving beyond the descriptive or conversational capacity measured by traditional scene understanding benchmarks. We provide a comprehensive suite of metrics to evaluate geometric accuracy, spatial relations, appearance attributes, and overall plausibility. Initial experiments on agentic inverse rendering powered by various state-of-the-art VLMs highlight current limitations, particularly in visual precision rather than basic tool usage. IR3D-Bench, including data and evaluation protocols, is released to facilitate systematic study and development of tool-using VLAs towards genuine scene understanding by creating.