Exploring the use of Generative AI to Support Automated Just-in-Time Programming for Visual Scene Displays

TL;DR

This study explores using large multimodal models like GPT-4V to automatically generate relevant communication options for visual scene displays, aiming to reduce manual configuration and improve AAC device support.

Contribution

It demonstrates the feasibility of leveraging LMMs to create contextually relevant communication options, comparing them with expert-generated options in AAC applications.

Findings

LMM-generated options were often contextually relevant and similar to human-created options.

Experts found the AI-generated options promising but identified key questions for deployment.

Study provides initial evidence supporting AI integration in AAC devices.

Abstract

Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options within contextualized images. However, existing VSDs often present default images that may lack relevance or require manual configuration, placing a significant burden on communication partners. In this study, we assess the feasibility of leveraging large multimodal models (LMM), such as GPT-4V, to automatically create communication options for VSDs. Communication options were sourced from a LMM and speech-language pathologists (SLPs) and AAC researchers (N=13) for evaluation through an expert assessment conducted by the SLPs and AAC researchers. We present the study's findings, supplemented by insights from semi-structured interviews (N=5) about SLP's and AAC researchers' opinions on the use of generative AI in augmentative and alternative communication devices. Our results indicate that the communication options generated by the LMM were contextually relevant and often resembled those created by humans. However, vital questions remain that must be addressed before LMMs can be confidently implemented in AAC devices.