Do Text-to-Vis Benchmarks Test Real Use of Visualisations?

TL;DR

This paper examines whether existing Text-to-Vis benchmarks accurately reflect real-world visualization use, revealing significant gaps and the need for more representative datasets to improve system development.

Contribution

It provides an empirical analysis comparing benchmark datasets with real-world code, highlighting discrepancies and proposing directions for creating more effective benchmarks.

Findings

Benchmarks do not match the distribution of real-world visualizations.

One dataset is somewhat representative but needs modifications.

New, more realistic benchmarks are necessary for progress.

Abstract

Large language models are able to generate code for visualisations in response to simple user requests. This is a useful application and an appealing one for NLP research because plots of data provide grounding for language. However, there are relatively few benchmarks, and those that exist may not be representative of what users do in practice. This paper investigates whether benchmarks reflect real-world use through an empirical study comparing benchmark datasets with code from public repositories. Our findings reveal a substantial gap, with evaluations not testing the same distribution of chart types, attributes, and actions as real-world examples. One dataset is representative, but requires extensive modification to become a practical end-to-end benchmark. This shows that new benchmarks are needed to support the development of systems that truly address users' visualisation needs. These observations will guide future data creation, highlighting which features hold genuine significance for users.