Empowering Dysarthric Speech: Leveraging Advanced LLMs for Accurate Speech Correction and Multimodal Emotion Analysis

TL;DR

This paper presents a novel framework that uses advanced large language models and speech recognition techniques to accurately correct dysarthric speech and analyze associated emotions, improving communication for affected individuals.

Contribution

It introduces an innovative combination of speech-to-text conversion, emotion detection, and sentence prediction using fine-tuned LLMs and benchmark models for dysarthric speech enhancement.

Findings

High accuracy in speech correction from dysarthric to intended sentences

Effective emotion recognition including happiness, sadness, and anger

Demonstrated scalability on benchmark models and specialized datasets

Abstract

Dysarthria is a motor speech disorder caused by neurological damage that affects the muscles used for speech production, leading to slurred, slow, or difficult-to-understand speech. It affects millions of individuals worldwide, including those with conditions such as stroke, traumatic brain injury, cerebral palsy, Parkinsons disease, and multiple sclerosis. Dysarthria presents a major communication barrier, impacting quality of life and social interaction. This paper introduces a novel approach to recognizing and translating dysarthric speech, empowering individuals with this condition to communicate more effectively. We leverage advanced large language models for accurate speech correction and multimodal emotion analysis. Dysarthric speech is first converted to text using OpenAI Whisper model, followed by sentence prediction using fine-tuned open-source models and benchmark models like GPT-4.o, LLaMA 3.1 70B and Mistral 8x7B on Groq AI accelerators. The dataset used combines the TORGO dataset with Google speech data, manually labeled for emotional context. Our framework identifies emotions such as happiness, sadness, neutrality, surprise, anger, and fear, while reconstructing intended sentences from distorted speech with high accuracy. This approach demonstrates significant advancements in the recognition and interpretation of dysarthric speech.