Automatic Assessment of Dysarthria Using Audio-visual Vowel Graph Attention Network

TL;DR

This paper introduces an audio-visual vowel graph attention network that combines expert acoustical features, deep learning representations, and visual cues to improve the automatic assessment of dysarthria, demonstrating superior regression performance.

Contribution

It presents a novel neural network architecture integrating expert knowledge, deep learning, and visual information for dysarthria assessment, enhancing interpretability and accuracy.

Findings

Outperforms existing methods in Frenchay score regression

Effectively combines acoustical features and deep learning representations

Incorporates visual information to improve robustness

Abstract

Automatic assessment of dysarthria remains a highly challenging task due to high variability in acoustic signals and the limited data. Currently, research on the automatic assessment of dysarthria primarily focuses on two approaches: one that utilizes expert features combined with machine learning, and the other that employs data-driven deep learning methods to extract representations. Research has demonstrated that expert features are effective in representing pathological characteristics, while deep learning methods excel at uncovering latent features. Therefore, integrating the advantages of expert features and deep learning to construct a neural network architecture based on expert knowledge may be beneficial for interpretability and assessment performance. In this context, the present paper proposes a vowel graph attention network based on audio-visual information, which effectively integrates the strengths of expert knowledges and deep learning. Firstly, various features were combined as inputs, including knowledge based acoustical features and deep learning based pre-trained representations. Secondly, the graph network structure based on vowel space theory was designed, allowing for a deep exploration of spatial correlations among vowels. Finally, visual information was incorporated into the model to further enhance its robustness and generalizability. The method exhibited superior performance in regression experiments targeting Frenchay scores compared to existing approaches.