Automatic Semantic Alignment of Flow Pattern Representations for Exploration with Large Language Models

TL;DR

This paper presents an automated framework that aligns flow pattern representations with large language model semantics, enabling natural language-based exploration and visualization of complex flow structures without manual labeling.

Contribution

The novel approach integrates autoencoder-encoded flow patterns with LLM embeddings, facilitating semantic matching and interactive natural language querying of flow data.

Findings

Effective semantic alignment between flow patterns and LLMs.

Enables natural language querying for flow visualization.

Improves accessibility of flow data analysis.

Abstract

Explorative flow visualization allows domain experts to analyze complex flow structures by interactively investigating flow patterns. However, traditional visual interfaces often rely on specialized graphical representations and interactions, which require additional effort to learn and use. Natural language interaction offers a more intuitive alternative, but teaching machines to recognize diverse scientific concepts and extract corresponding structures from flow data poses a significant challenge. In this paper, we introduce an automated framework that aligns flow pattern representations with the semantic space of large language models (LLMs), eliminating the need for manual labeling. Our approach encodes streamline segments using a denoising autoencoder and maps the generated flow pattern representations to LLM embeddings via a projector layer. This alignment empowers semantic matching between textual embeddings and flow representations through an attention mechanism, enabling the extraction of corresponding flow patterns based on textual descriptions. To enhance accessibility, we develop an interactive interface that allows users to query and visualize flow structures using natural language. Through case studies, we demonstrate the effectiveness of our framework in enabling intuitive and intelligent flow exploration.