Audio-Vision Contrastive Learning for Phonological Class Recognition

TL;DR

This paper introduces a contrastive learning framework combining rtMRI and speech signals for classifying phonological features, achieving state-of-the-art results in articulatory analysis.

Contribution

It presents a novel multimodal contrastive learning approach that enhances phonological class recognition using real-time MRI and audio data.

Findings

Contrastive learning improves classification accuracy.

State-of-the-art F1-score of 0.81 achieved.

Multimodal fusion outperforms unimodal baselines.

Abstract

Accurate classification of articulatory-phonological features plays a vital role in understanding human speech production and developing robust speech technologies, particularly in clinical contexts where targeted phonemic analysis and therapy can improve disease diagnosis accuracy and personalized rehabilitation. In this work, we propose a multimodal deep learning framework that combines real-time magnetic resonance imaging (rtMRI) and speech signals to classify three key articulatory dimensions: manner of articulation, place of articulation, and voicing. We perform classification on 15 phonological classes derived from the aforementioned articulatory dimensions and evaluate the system with four audio/vision configurations: unimodal rtMRI, unimodal audio signals, multimodal middle fusion, and contrastive learning-based audio-vision fusion. Experimental results on the USC-TIMIT dataset show that our contrastive learning-based approach achieves state-of-the-art performance, with an average F1-score of 0.81, representing an absolute increase of 0.23 over the unimodal baseline. The results confirm the effectiveness of contrastive representation learning for multimodal articulatory analysis. Our code and processed dataset will be made publicly available at https://github.com/DaE-plz/AC_Contrastive_Phonology to support future research.