Re-Thinking Inverse Graphics With Large Language Models

TL;DR

This paper introduces IG-LLM, a novel framework that leverages large language models to perform inverse graphics tasks by decoding visual embeddings into structured 3D scene representations, enabling zero-shot generalization without image-space supervision.

Contribution

The paper presents IG-LLM, a pioneering approach that uses LLMs for inverse graphics, combining visual encoders with language models for structured 3D scene understanding.

Findings

LLMs can decode visual embeddings into 3D scene representations.

The approach enables zero-shot inverse graphics without image supervision.

End-to-end training with a frozen visual encoder is effective.

Abstract

Inverse graphics -- the task of inverting an image into physical variables that, when rendered, enable reproduction of the observed scene -- is a fundamental challenge in computer vision and graphics. Successfully disentangling an image into its constituent elements, such as the shape, color, and material properties of the objects of the 3D scene that produced it, requires a comprehensive understanding of the environment. This complexity limits the ability of existing carefully engineered approaches to generalize across domains. Inspired by the zero-shot ability of large language models (LLMs) to generalize to novel contexts, we investigate the possibility of leveraging the broad world knowledge encoded in such models to solve inverse-graphics problems. To this end, we propose the Inverse-Graphics Large Language Model (IG-LLM), an inverse-graphics framework centered around an LLM, that autoregressively decodes a visual embedding into a structured, compositional 3D-scene representation. We incorporate a frozen pre-trained visual encoder and a continuous numeric head to enable end-to-end training. Through our investigation, we demonstrate the potential of LLMs to facilitate inverse graphics through next-token prediction, without the application of image-space supervision. Our analysis enables new possibilities for precise spatial reasoning about images that exploit the visual knowledge of LLMs. We release our code and data at https://ig-llm.is.tue.mpg.de/ to ensure the reproducibility of our investigation and to facilitate future research.