Natural Language-Driven Viewpoint Navigation for Volume Exploration via Semantic Block Representation

TL;DR

This paper introduces a natural language-driven framework for volumetric data exploration that uses semantic block encoding, CLIP Score, and reinforcement learning to automate viewpoint selection and improve interpretability.

Contribution

It presents a novel integration of natural language interaction, semantic encoding, and reinforcement learning for efficient volumetric data navigation.

Findings

Enhanced navigation efficiency demonstrated in experiments.

Semantic block representation improves structure differentiation.

Automated viewpoint selection aligns with user intent.

Abstract

Exploring volumetric data is crucial for interpreting scientific datasets. However, selecting optimal viewpoints for effective navigation can be challenging, particularly for users without extensive domain expertise or familiarity with 3D navigation. In this paper, we propose a novel framework that leverages natural language interaction to enhance volumetric data exploration. Our approach encodes volumetric blocks to capture and differentiate underlying structures. It further incorporates a CLIP Score mechanism, which provides semantic information to the blocks to guide navigation. The navigation is empowered by a reinforcement learning framework that leverage these semantic cues to efficiently search for and identify desired viewpoints that align with the user's intent. The selected viewpoints are evaluated using CLIP Score to ensure that they best reflect the user queries. By automating viewpoint selection, our method improves the efficiency of volumetric data navigation and enhances the interpretability of complex scientific phenomena.