Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language

TL;DR

This paper enhances Gujarati-English code-switching speech recognition by integrating language identification into transformer models, improving language prediction accuracy despite limited WER reduction.

Contribution

It introduces novel methods for conditioning transformer layers on language ID and character, along with a Temporal Loss for better input alignment in code-switched speech recognition.

Findings

Improved language prediction accuracy from spoken data

Proposed methods for language-specific parameters and explainability in attention

Regularization techniques aiding sequence alignment

Abstract

An important and difficult task in code-switched speech recognition is to recognize the language, as lots of words in two languages can sound similar, especially in some accents. We focus on improving performance of end-to-end Automatic Speech Recognition models by conditioning transformer layers on language ID of words and character in the output in an per layer supervised manner. To this end, we propose two methods of introducing language specific parameters and explainability in the multi-head attention mechanism, and implement a Temporal Loss that helps maintain continuity in input alignment. Despite being unable to reduce WER significantly, our method shows promise in predicting the correct language from just spoken data. We introduce regularization in the language prediction by dropping LID in the sequence, which helps align long repeated output sequences.