NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splatting

TL;DR

NLI4VolVis is an interactive system that uses natural language and multi-agent LLMs to enable semantic exploration, editing, and visualization of volumetric scenes with 3D Gaussian splatting, improving accessibility and usability.

Contribution

It introduces a novel multi-agent LLM architecture with function-calling tools for semantic interaction and editing in volume visualization using 3D Gaussian splatting.

Findings

Enhanced semantic querying and editing capabilities in volumetric scenes.

Improved user accessibility and interaction efficiency demonstrated through case studies.

Open-vocabulary object querying and real-time scene editing achieved.

Abstract

Traditional volume visualization (VolVis) methods, like direct volume rendering, suffer from rigid transfer function designs and high computational costs. Although novel view synthesis approaches enhance rendering efficiency, they require additional learning effort for non-experts and lack support for semantic-level interaction. To bridge this gap, we propose NLI4VolVis, an interactive system that enables users to explore, query, and edit volumetric scenes using natural language. NLI4VolVis integrates multi-view semantic segmentation and vision-language models to extract and understand semantic components in a scene. We introduce a multi-agent large language model architecture equipped with extensive function-calling tools to interpret user intents and execute visualization tasks. The agents leverage external tools and declarative VolVis commands to interact with the VolVis engine powered by 3D editable Gaussians, enabling open-vocabulary object querying, real-time scene editing, best-view selection, and 2D stylization. We validate our system through case studies and a user study, highlighting its improved accessibility and usability in volumetric data exploration. We strongly recommend readers check our case studies, demo video, and source code at https://nli4volvis.github.io/.