Semantic Scaffolding: Augmenting Textual Structures with Domain-Specific Groupings for Accessible Data Exploration

TL;DR

Semantic scaffolding leverages large language models to create domain-specific groupings and annotations in data, enhancing accessibility and understanding for users with varying expertise.

Contribution

This paper introduces a novel technique using LLMs to generate semantic groupings and explanations, improving data exploration for visually impaired users.

Findings

BLV users understood data better with semantic scaffolds

Users were aware of the influence of scaffolds on interpretation

Semantic scaffolding facilitated quicker data comprehension

Abstract

Drawing connections between interesting groupings of data and their real-world meaning is an important, yet difficult, part of encountering a new dataset. A lay reader might see an interesting visual pattern in a chart but lack the domain expertise to explain its meaning. Or, a reader might be familiar with a real-world concept but struggle to express it in terms of a dataset's fields. In response, we developed semantic scaffolding, a technique for using domain-specific information from large language models (LLMs) to identify, explain, and formalize semantically meaningful data groupings. We present groupings in two ways: as semantic bins, which segment a field into domain-specific intervals and categories; and data highlights, which annotate subsets of data records with their real-world meaning. We demonstrate and evaluate this technique in Olli, an accessible visualization tool that exemplifies tensions around explicitly defining groupings while respecting the agency of readers to conduct independent data exploration. We conducted a study with 15 blind and low-vision (BLV) users and found that readers used semantic scaffolds to quickly understand the meaning of the data, but were often also critically aware of its influence on their interpretation.