ImageTalk: Designing a Multimodal AAC Text Generation System Driven by Image Recognition and Natural Language Generation

TL;DR

ImageTalk is a multimodal AAC system that combines image recognition and natural language generation to significantly improve communication efficiency for people with Motor Neuron Disease, achieving high user satisfaction and keystroke savings.

Contribution

The paper introduces a novel multimodal AAC text generation system called ImageTalk, designed through user-centered methods, with guidelines and requirements for future AI-assisted AAC systems.

Findings

95.6% keystroke savings achieved

High user satisfaction reported

Consistent performance across users

Abstract

People living with Motor Neuron Disease (plwMND) frequently encounter speech and motor impairments that necessitate a reliance on augmentative and alternative communication (AAC) systems. This paper tackles the main challenge that traditional symbol-based AAC systems offer a limited vocabulary, while text entry solutions tend to exhibit low communication rates. To help plwMND articulate their needs about the system efficiently and effectively, we iteratively design and develop a novel multimodal text generation system called ImageTalk through a tailored proxy-user-based and an end-user-based design phase. The system demonstrates pronounced keystroke savings of 95.6%, coupled with consistent performance and high user satisfaction. We distill three design guidelines for AI-assisted text generation systems design and outline four user requirement levels tailored for AAC purposes, guiding future research in this field.