Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition

TL;DR

This paper introduces Cued-Agent, a novel collaborative multi-agent system that significantly improves automatic cued speech recognition by integrating specialized agents for hand and lip recognition, dynamic prompt decoding, and semantic refinement, especially under limited data conditions.

Contribution

It presents the first multi-agent system for ACSR, combining multiple specialized agents to enhance recognition accuracy and robustness in limited data scenarios.

Findings

Outperforms state-of-the-art methods in recognition accuracy.

Effectively handles multimodal fusion with limited data.

Enables end-to-end phoneme-to-word conversion.

Abstract

Cued Speech (CS) is a visual communication system that combines lip-reading with hand coding to facilitate communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) aims to convert CS hand gestures and lip movements into text via AI-driven methods. Traditionally, the temporal asynchrony between hand and lip movements requires the design of complex modules to facilitate effective multimodal fusion. However, constrained by limited data availability, current methods demonstrate insufficient capacity for adequately training these fusion mechanisms, resulting in suboptimal performance. Recently, multi-agent systems have shown promising capabilities in handling complex tasks with limited data availability. To this end, we propose the first collaborative multi-agent system for ACSR, named Cued-Agent. It integrates four specialized sub-agents: a Multimodal Large Language Model-based Hand Recognition agent that employs keyframe screening and CS expert prompt strategies to decode hand movements, a pretrained Transformer-based Lip Recognition agent that extracts lip features from the input video, a Hand Prompt Decoding agent that dynamically integrates hand prompts with lip features during inference in a training-free manner, and a Self-Correction Phoneme-to-Word agent that enables post-process and end-to-end conversion from phoneme sequences to natural language sentences for the first time through semantic refinement. To support this study, we expand the existing Mandarin CS dataset by collecting data from eight hearing-impaired cuers, establishing a mixed dataset of fourteen subjects. Extensive experiments demonstrate that our Cued-Agent performs superbly in both normal and hearing-impaired scenarios compared with state-of-the-art methods. The implementation is available at https://github.com/DennisHgj/Cued-Agent.