Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text

TL;DR

Text2Vis introduces a comprehensive benchmark for evaluating text-to-visualization models across diverse chart types and data queries, highlighting current challenges and proposing a novel agent-critic framework to improve model performance.

Contribution

The paper presents the first extensive benchmark for text-to-visualization, along with a new cross-modal agent-critic framework and automated evaluation methods to advance the field.

Findings

Significant performance gaps among existing models.

The proposed framework improves visualization quality and answer accuracy.

Automated evaluation enables scalable assessment without human annotation.

Abstract

Automated data visualization plays a crucial role in simplifying data interpretation, enhancing decision-making, and improving efficiency. While large language models (LLMs) have shown promise in generating visualizations from natural language, the absence of comprehensive benchmarks limits the rigorous evaluation of their capabilities. We introduce Text2Vis, a benchmark designed to assess text-to-visualization models, covering 20+ chart types and diverse data science queries, including trend analysis, correlation, outlier detection, and predictive analytics. It comprises 1,985 samples, each with a data table, natural language query, short answer, visualization code, and annotated charts. The queries involve complex reasoning, conversational turns, and dynamic data retrieval. We benchmark 11 open-source and closed-source models, revealing significant performance gaps, highlighting key challenges, and offering insights for future advancements. To close this gap, we propose the first cross-modal actor-critic agentic framework that jointly refines the textual answer and visualization code, increasing GPT-4o`s pass rate from 26% to 42% over the direct approach and improving chart quality. We also introduce an automated LLM-based evaluation framework that enables scalable assessment across thousands of samples without human annotation, measuring answer correctness, code execution success, visualization readability, and chart accuracy. We release Text2Vis at https://github.com/vis-nlp/Text2Vis.