Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition

TL;DR

This study introduces a transformer-based multimodal framework combining acoustic and textual data for Arabic phoneme mispronunciation detection, significantly improving accuracy in Quranic recitation analysis.

Contribution

It presents a novel multimodal deep learning approach integrating UniSpeech and BERT embeddings with various fusion strategies for enhanced phoneme mispronunciation detection.

Findings

UniSpeech-BERT configuration yields high accuracy

Fusion strategies improve detection robustness

Model generalizes well across diverse datasets

Abstract

Recent advances in multimodal deep learning have greatly enhanced the capability of systems for speech analysis and pronunciation assessment. Accurate pronunciation detection remains a key challenge in Arabic, particularly in the context of Quranic recitation, where subtle phonetic differences can alter meaning. Addressing this challenge, the present study proposes a transformer-based multimodal framework for Arabic phoneme mispronunciation detection that combines acoustic and textual representations to achieve higher precision and robustness. The framework integrates UniSpeech-derived acoustic embeddings with BERT-based textual embeddings extracted from Whisper transcriptions, creating a unified representation that captures both phonetic detail and linguistic context. To determine the most effective integration strategy, early, intermediate, and late fusion methods were implemented and evaluated on two datasets containing 29 Arabic phonemes, including eight hafiz sounds, articulated by 11 native speakers. Additional speech samples collected from publicly available YouTube recordings were incorporated to enhance data diversity and generalization. Model performance was assessed using standard evaluation metrics: accuracy, precision, recall, and F1-score, allowing a detailed comparison of the fusion strategies. Experimental findings show that the UniSpeech-BERT multimodal configuration provides strong results and that fusion-based transformer architectures are effective for phoneme-level mispronunciation detection. The study contributes to the development of intelligent, speaker-independent, and multimodal Computer-Aided Language Learning (CALL) systems, offering a practical step toward technology-supported Quranic pronunciation training and broader speech-based educational applications.