# Leveraging native language information for improved accented speech recognition

## TL;DR

This paper investigates how incorporating native language data into end-to-end RNN models can significantly improve accented English speech recognition, especially for bilingual speakers with Hispanic and Indian accents.

## Contribution

It introduces a multi-task learning approach that leverages native language data alongside English to enhance accented speech recognition performance.

## Key findings

- Multi-task learning outperforms pre-training methods.
- Proposed MTL setting with combined native and English data yields better results.
- Achieved +11.95% and +17.55% CER improvements for Hispanic and Indian accents.

## Abstract

Recognition of accented speech is a long-standing challenge for automatic speech recognition (ASR) systems, given the increasing worldwide population of bi-lingual speakers with English as their second language. If we consider foreign-accented speech as an interpolation of the native language (L1) and English (L2), using a model that can simultaneously address both languages would perform better at the acoustic level for accented speech. In this study, we explore how an end-to-end recurrent neural network (RNN) trained system with English and native languages (Spanish and Indian languages) could leverage data of native languages to improve performance for accented English speech. To this end, we examine pre-training with native languages, as well as multi-task learning (MTL) in which the main task is trained with native English and the secondary task is trained with Spanish or Indian Languages. We show that the proposed MTL model performs better than the pre-training approach and outperforms a baseline model trained simply with English data. We suggest a new setting for MTL in which the secondary task is trained with both English and the native language, using the same output set. This proposed scenario yields better performance with +11.95% and +17.55% character error rate gains over baseline for Hispanic and Indian accents, respectively.