Giving Meaning to Movements: Challenges and Opportunities in Expanding Communication by Pairing Unaided AAC with Speech Generated Messages

TL;DR

This paper explores combining unaided and aided AAC methods through a wearable system, AllyAAC, to improve naturalness and intelligibility in communication for users with impairments, supported by a large multimodal dataset and ML models.

Contribution

It introduces AllyAAC, a novel wearable system that integrates unaided gestures with speech-generating messages, and provides design principles and a dataset for personalized gesture recognition.

Findings

Transformer-based ML models improve gesture recognition accuracy.

Personalized gestures pose recognition challenges addressed by pretraining strategies.

The dataset is the first of its kind with over 600,000 multimodal data points.

Abstract

Augmentative and Alternative Communication (AAC) technologies are categorized into two forms: aided AAC, which uses external devices like speech-generating systems to produce standardized output, and unaided AAC, which relies on body-based gestures for natural expression but requires shared understanding. We investigate how to combine these approaches to harness the speed and naturalness of unaided AAC while maintaining the intelligibility of aided AAC, a largely unexplored area for individuals with communication and motor impairments. Through 18 months of participatory design with AAC users, we identified key challenges and opportunities and developed AllyAAC, a wearable system with a wrist-worn IMU paired with a smartphone app. We evaluated AllyAAC in a field study with 14 participants and produced a dataset containing over 600,000 multimodal data points featuring atypical gestures--the first of its kind. Our findings reveal challenges in recognizing personalized, idiosyncratic gestures and demonstrate how to address them using Transformer-based large machine learning (ML) models with different pretraining strategies. In sum, we contribute design principles and a reference implementation for adaptive, personalized systems combining aided and unaided AAC.