Semantic Scaffolding: Augmenting Textual Structures with Domain-Specific Groupings for Accessible Data Exploration
Jonathan Zong, Isabella Pedraza Pineros, Mengzhu Katie Chen, Daniel Hajas, Arvind Satyanarayan

TL;DR
Semantic scaffolding leverages large language models to create domain-specific groupings and annotations in data, enhancing accessibility and understanding for users with varying expertise.
Contribution
This paper introduces a novel technique using LLMs to generate semantic groupings and explanations, improving data exploration for visually impaired users.
Findings
BLV users understood data better with semantic scaffolds
Users were aware of the influence of scaffolds on interpretation
Semantic scaffolding facilitated quicker data comprehension
Abstract
Drawing connections between interesting groupings of data and their real-world meaning is an important, yet difficult, part of encountering a new dataset. A lay reader might see an interesting visual pattern in a chart but lack the domain expertise to explain its meaning. Or, a reader might be familiar with a real-world concept but struggle to express it in terms of a dataset's fields. In response, we developed semantic scaffolding, a technique for using domain-specific information from large language models (LLMs) to identify, explain, and formalize semantically meaningful data groupings. We present groupings in two ways: as semantic bins, which segment a field into domain-specific intervals and categories; and data highlights, which annotate subsets of data records with their real-world meaning. We demonstrate and evaluate this technique in Olli, an accessible visualization tool that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Visualization and Analytics · Multimodal Machine Learning Applications · Scientific Computing and Data Management
