Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation

TL;DR

This paper introduces a graph neural network framework that models speech and gesture interactions to estimate aphasia severity, providing a more holistic and automated assessment method compared to traditional isolated linguistic features.

Contribution

It presents a novel graph-based approach that captures structured speech-gesture interactions for more accurate aphasia severity estimation.

Findings

Structured speech-gesture interactions better predict aphasia severity.

Graph neural networks outperform isolated feature-based models.

Potential for bedside and telehealth aphasia assessment.

Abstract

Aphasia is an acquired language disorder caused by injury to the regions of the brain that are responsible for language. Aphasia may impair the use and comprehension of written and spoken language. The Western Aphasia Battery-Revised (WAB-R) is an assessment tool administered by speech-language pathologists (SLPs) to evaluate the aphasia type and severity. Because the WAB-R measures isolated linguistic skills, there has been growing interest in the assessment of discourse production as a more holistic representation of everyday language abilities. Recent advancements in speech analysis focus on automated estimation of aphasia severity from spontaneous speech, relying mostly in isolated linguistic or acoustical features. In this work, we propose a graph neural network-based framework for estimating aphasia severity. We represented each participant's discourse as a directed multi-modal graph, where nodes represent lexical items and gestures and edges encode word-word, gesture-word, and word-gesture transitions. GraphSAGE is employed to learn participant-level embeddings, thus integrating information from immediate neighbors and overall graph structure. Our results suggest that aphasia severity is not encoded in isolated lexical distribution, but rather emerges from structured interactions between speech and gesture. The proposed architecture offers a reliable automated aphasia assessment, with possible uses in bedside screening and telehealth-based monitoring.