VegaChat: A Robust Framework for LLM-Based Chart Generation and Assessment

TL;DR

VegaChat is a comprehensive framework that improves the generation and evaluation of data visualizations from natural language using new metrics and LLMs, enhancing consistency and comparability.

Contribution

It introduces two novel evaluation metrics, Spec Score and Vision Score, for assessing visualization quality and compliance, addressing key challenges in NL2VIS systems.

Findings

VegaChat achieves near-zero invalid visualizations.

Spec Score and Vision Score strongly correlate with human judgments.

The framework enables consistent, cross-library comparison of visualizations.

Abstract

Natural-language-to-visualization (NL2VIS) systems based on large language models (LLMs) have substantially improved the accessibility of data visualization. However, their further adoption is hindered by two coupled challenges: (i) the absence of standardized evaluation metrics makes it difficult to assess progress in the field and compare different approaches; and (ii) natural language descriptions are inherently underspecified, so multiple visualizations may be valid for the same query. To address these issues, we introduce VegaChat, a framework for generating, validating, and assessing declarative visualizations from natural language. We propose two complementary metrics: Spec Score, a deterministic metric that measures specification-level similarity without invoking an LLM, and Vision Score, a library-agnostic, image-based metric that leverages a multimodal LLM to assess chart similarity and prompt compliance. We evaluate VegaChat on the NLV Corpus and on the annotated subset of ChartLLM. VegaChat achieves near-zero rates of invalid or empty visualizations, while Spec Score and Vision Score exhibit strong correlation with human judgments (Pearson 0.65 and 0.71, respectively), indicating that the proposed metrics support consistent, cross-library comparison. The code and evaluation artifacts are available at https://zenodo.org/records/17062309.