Supporting Multimodal Data Interaction on Refreshable Tactile Displays: An Architecture to Combine Touch and Conversational AI

TL;DR

This paper introduces a novel architecture that combines touch input and conversational AI on refreshable tactile displays to enhance accessible data visualization for visually impaired users, supported by an open-source implementation.

Contribution

It presents the first system integrating touch sensing with conversational AI on RTDs, addressing key technical challenges and providing a foundation for future multimodal accessibility research.

Findings

First to combine touch and conversational AI on RTDs

Demonstrates feasibility with an open-source implementation

Enables deictic queries using touch context and speech

Abstract

Combining conversational AI with refreshable tactile displays (RTDs) offers significant potential for creating accessible data visualization for people who are blind or have low vision (BLV). To support researchers and developers building accessible data visualizations with RTDs, we present a multimodal data interaction architecture along with an open-source reference implementation. Our system is the first to combine touch input with a conversational agent on an RTD, enabling deictic queries that fuse touch context with spoken language, such as "what is the trend between these points?" The architecture addresses key technical challenges, including touch sensing on RTDs, visual-to-tactile encoding, integrating touch context with conversational AI, and synchronizing multimodal output. Our contributions are twofold: (1) a technical architecture integrating RTD hardware, external touch sensing, and conversational AI to enable multimodal data interaction; and (2) an open-source reference implementation demonstrating its feasibility. This work provides a technical foundation to support future research in multimodal accessible data visualization.