SpeechAgent: An End-to-End Mobile Infrastructure for Speech Impairment Assistance

TL;DR

SpeechAgent is a mobile infrastructure that combines advanced speech processing and large language models to provide real-time, personalized communication support for individuals with speech impairments, enabling practical daily use.

Contribution

The paper introduces SpeechAgent, a novel mobile system integrating LLM-driven reasoning with speech processing for assistive communication, optimized for real-world deployment.

Findings

Effective real-time speech processing on mobile devices

High accuracy and speech quality in impaired speech datasets

Low latency performance suitable for daily use

Abstract

Speech is essential for human communication, yet millions of people face impairments such as dysarthria, stuttering, and aphasia conditions that often lead to social isolation and reduced participation. Despite recent progress in automatic speech recognition (ASR) and text-to-speech (TTS) technologies, accessible web and mobile infrastructures for users with impaired speech remain limited, hindering the practical adoption of these advances in daily communication. To bridge this gap, we present SpeechAgent, a mobile SpeechAgent designed to facilitate people with speech impairments in everyday communication. The system integrates large language model (LLM)- driven reasoning with advanced speech processing modules, providing adaptive support tailored to diverse impairment types. To ensure real-world practicality, we develop a structured deployment pipeline that enables real-time speech processing on mobile and edge devices, achieving imperceptible latency while maintaining high accuracy and speech quality. Evaluation on real-world impaired speech datasets and edge-device latency profiling confirms that SpeechAgent delivers both effective and user-friendly performance, demonstrating its feasibility for personalized, day-to-day assistive communication.