AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making

TL;DR

This paper introduces Autonomous Visualization Agents (AVAs) that leverage visual perception in multi-modal LLMs to interpret and achieve user-defined visualization goals, enabling domain experts to generate visualizations through natural language.

Contribution

It presents the first framework for AVAs utilizing visual perception in multi-modal LLMs, demonstrating their applicability across various visualization tasks and scenarios.

Findings

AVAs can interpret visual outputs to assist in visualization tasks.

Preliminary agents show potential in domain-specific visualization applications.

Expert feedback indicates high practicality and future potential of AVAs.

Abstract

With recent advances in multi-modal foundation models, the previously text-only large language models (LLM) have evolved to incorporate visual input, opening up unprecedented opportunities for various applications in visualization. Our work explores the utilization of the visual perception ability of multi-modal LLMs to develop Autonomous Visualization Agents (AVAs) that can interpret and accomplish user-defined visualization objectives through natural language. We propose the first framework for the design of AVAs and present several usage scenarios intended to demonstrate the general applicability of the proposed paradigm. The addition of visual perception allows AVAs to act as the virtual visualization assistant for domain experts who may lack the knowledge or expertise in fine-tuning visualization outputs. Our preliminary exploration and proof-of-concept agents suggest that this approach can be widely applicable whenever the choices of appropriate visualization parameters require the interpretation of previous visual output. Feedback from unstructured interviews with experts in AI research, medical visualization, and radiology has been incorporated, highlighting the practicality and potential of AVAs. Our study indicates that AVAs represent a general paradigm for designing intelligent visualization systems that can achieve high-level visualization goals, which pave the way for developing expert-level visualization agents in the future.