LLaNA: Large Language and NeRF Assistant

TL;DR

LLaNA introduces a novel approach to integrate Neural Radiance Fields with large language models, enabling direct understanding and interaction with 3D object representations without rendering images.

Contribution

This work is the first to ingest NeRF weights into a language model, creating a general-purpose NeRF-language assistant capable of new tasks like captioning and Q&A.

Findings

Processing NeRF weights outperforms 2D/3D extraction methods.

Built a NeRF dataset with text annotations for benchmarking.

Demonstrated effective NeRF understanding without rendering.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated an excellent understanding of images and 3D data. However, both modalities have shortcomings in holistically capturing the appearance and geometry of objects. Meanwhile, Neural Radiance Fields (NeRFs), which encode information within the weights of a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly widespread modality that simultaneously encodes the geometry and photorealistic appearance of objects. This paper investigates the feasibility and effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first general-purpose NeRF-language assistant capable of performing new tasks such as NeRF captioning and Q\&A. Notably, our method directly processes the weights of the NeRF's MLP to extract information about the represented objects without the need to render images or materialize 3D data structures. Moreover, we build a dataset of NeRFs with text annotations for various NeRF-language tasks with no human intervention. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that processing NeRF weights performs favourably against extracting 2D or 3D representations from NeRFs.