Whisper based Cross-Lingual Phoneme Recognition between Vietnamese and English

TL;DR

This paper introduces a novel bilingual speech recognition system leveraging Whisper's pre-trained encoder to improve cross-lingual phoneme recognition between Vietnamese and English, addressing tonal and stress pattern challenges.

Contribution

It proposes a new bilingual phoneme set and an end-to-end recognition system using PhoWhisper encoder, advancing cross-lingual phoneme recognition techniques.

Findings

Improved recognition accuracy for Vietnamese-English bilingual speech.

Robust framework for tonal and stress-based phoneme recognition.

Effective bridging of phonetic differences between Vietnamese and English.

Abstract

Cross-lingual phoneme recognition has emerged as a significant challenge for accurate automatic speech recognition (ASR) when mixing Vietnamese and English pronunciations. Unlike many languages, Vietnamese relies on tonal variations to distinguish word meanings, whereas English features stress patterns and non-standard pronunciations that hinder phoneme alignment between the two languages. To address this challenge, we propose a novel bilingual speech recognition approach with two primary contributions: (1) constructing a representative bilingual phoneme set that bridges the differences between Vietnamese and English phonetic systems; (2) designing an end-to-end system that leverages the PhoWhisper pre-trained encoder for deep high-level representations to improve phoneme recognition. Our extensive experiments demonstrate that the proposed approach not only improves recognition accuracy in bilingual speech recognition for Vietnamese but also provides a robust framework for addressing the complexities of tonal and stress-based phoneme recognition