Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning

TL;DR

VIGA is a training-free, multimodal reasoning framework that reconstructs images into editable programs through an iterative code-render-inspect loop, enhancing accuracy across diverse visual tasks.

Contribution

Introduces VIGA, a novel, training-free framework for vision-as-inverse-graphics using interleaved multimodal reasoning and a new benchmark, BlenderBench.

Findings

VIGA improves accuracy significantly over one-shot baselines.

VIGA supports diverse tasks like 2D document generation and 3D reconstruction.

Empirical results show substantial accuracy gains on multiple benchmarks.

Abstract

Vision-as-inverse-graphics, the concept of reconstructing images into editable programs, remains challenging for Vision-Language Models (VLMs), which inherently lack fine-grained spatial grounding in one-shot settings. To address this, we introduce VIGA (Vision-as-Inverse-Graphics Agent), an interleaved multimodal reasoning framework where symbolic logic and visual perception actively cross-verify each other. VIGA operates through a tightly coupled code-render-inspect loop: synthesizing symbolic programs, projecting them into visual states, and inspecting discrepancies to guide iterative edits. Equipped with high-level semantic skills and an evolving multimodal memory, VIGA sustains evidence-based modifications over long horizons. This training-free, task-agnostic framework seamlessly supports 2D document generation, 3D reconstruction, multi-step 3D editing, and 4D physical interaction. Finally, we introduce BlenderBench, a challenging visual-to-code benchmark. Empirically, VIGA substantially improves accuracy compared with one-shot baselines in BlenderGym (35.32%), SlideBench (117.17%) and our proposed BlenderBench (124.70%).