HI-TransPA: Hearing Impairments Translation Personal Assistant

TL;DR

HI-TransPA is a multimodal assistive system that translates and interprets hearing-impaired speech by combining audio and lip movement analysis, improving communication for the hearing-impaired.

Contribution

The paper introduces a novel Omni-Model framework with a unified 3D-Resampler for assistive tech, along with a quality-guided curriculum learning strategy for hearing-impaired speech translation.

Findings

Achieves state-of-the-art accuracy and semantic fidelity on HI-Dialogue dataset.

Develops a multimodal preprocessing pipeline for lip and facial landmark detection.

Demonstrates robustness through curriculum learning with sample quality scores.

Abstract

Hearing-impaired individuals often face significant barriers in daily communication due to the inherent challenges of producing clear speech. To address this, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework. To address the distinctive pronunciation patterns of hearing-impaired speech and the limited adaptability of existing models, we develop a multimodal preprocessing and curation pipeline that detects facial landmarks, stabilizes the lip region, and quantitatively evaluates sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. Architecturally, we employs a novel unified 3D-Resampler to efficiently encode the lip dynamics, which is critical for accurate interpretation. Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. Our work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research.