State of the Art of LLM-Enabled Interaction with Visualization

TL;DR

This survey reviews how large language models are integrated with visualization tools to enable natural language interaction, highlighting current research, design patterns, and gaps for future development.

Contribution

It provides a systematic classification framework for LLM-visualization systems, analyzing 48 papers and identifying key design patterns and research gaps.

Findings

LLMs enable intuitive multimodal data interaction.

Current systems face challenges in spatial reasoning and contextual understanding.

There are significant gaps in accessibility and evaluation methods.

Abstract

We report on a systematic, PRISMA-guided survey of research at the intersection of LLMs and visualization, with a particular focus on visio-verbal interaction -- where verbal and visual modalities converge to support data sense-making. The emergence of Large Language Models (LLMs) has introduced new paradigms for interacting with data visualizations through natural language, leading to intuitive, multimodal, and accessible interfaces. We analyze 48 papers across six dimensions: application domain, visualization task, visualization representation, interaction modality, LLM integration, and system evaluation. Our classification framework maps LLM roles across the visualization pipeline, from data querying and transformation to visualization generation, explanation, and navigation. We highlight emerging design patterns, identify gaps in accessibility and visualization reading, and discuss the limitations of current LLMs in spatial reasoning and contextual grounding. We further reflect on evaluations of combined LLM-visualization systems, highlighting how current research projects tackle this challenge and discuss current gaps in conducting meaningful evaluations of such systems. With our survey we aim to guide future research and system design in LLM-enhanced visualization, supporting broad audiences and intelligent, conversational interfaces.