Talk Me Through It: Developing Effective Systems for Chart Authoring

TL;DR

This paper investigates the differences in command structures between spoken and typed chart creation instructions and demonstrates that training systems on spoken imagined-chart data significantly improves their performance.

Contribution

It provides empirical analysis of instruction differences and shows the benefits of training on spoken imagined-chart data for better chart-authoring systems.

Findings

Spoken imagined-chart instructions have richer command structures.

Systems trained on spoken data outperform those trained on typed data.

Design guidelines are proposed for real-world chart-authoring systems.

Abstract

Recent chart-authoring systems increasingly focus on natural-language input, enabling users to form a mental image of the chart they wish to create and express this intent using spoken instructions (spoken imagined-chart data). Yet these systems are predominantly trained on typed instructions written while viewing the target chart (typed existing-chart data). While the cognitive processes for describing an existing chart arguably differ from those for creating a new chart, the structural differences in the corresponding prompts remain underexplored. We present empirical findings on the structural differences among spoken imagined-chart instructions, typed imagined-chart instructions, and typed existing-chart instructions for chart creation, showing that imagined-chart prompts contain richer command formats, element specifications, and complex linguistic features, especially in spoken instructions. We then compare the performance of systems trained on spoken imagined-chart data versus typed existing-chart data, finding that the first system outperforms the second one on both voice and text input, highlighting the necessity of targeted training on spoken imagined-chart data. We conclude with design guidelines for chart-authoring systems to improve performance in real-world scenarios.