Multi-Accent Adaptation based on Gate Mechanism

TL;DR

This paper introduces a multi-accent speech recognition method using a gate mechanism that adapts models to multiple accents simultaneously, improving recognition accuracy especially when accent labels are unavailable at inference.

Contribution

The paper proposes the AST-G and MTL-G models for multi-accent adaptation, with AST-G using a gate mechanism for simultaneous adaptation and MTL-G integrating an accent classifier through multi-task learning.

Findings

AST-G achieves 9.8% and 1.9% relative WER reduction over baseline and accent-specific models.

MTL-G achieves 5.1% relative WER reduction over baseline, despite less accuracy than accent-specific adaptation.

Using a gate mechanism effectively improves multi-accent speech recognition performance.

Abstract

When only a limited amount of accented speech data is available, to promote multi-accent speech recognition performance, the conventional approach is accent-specific adaptation, which adapts the baseline model to multiple target accents independently. To simplify the adaptation procedure, we explore adapting the baseline model to multiple target accents simultaneously with multi-accent mixed data. Thus, we propose using accent-specific top layer with gate mechanism (AST-G) to realize multi-accent adaptation. Compared with the baseline model and accent-specific adaptation, AST-G achieves 9.8% and 1.9% average relative WER reduction respectively. However, in real-world applications, we can't obtain the accent category label for inference in advance. Therefore, we apply using an accent classifier to predict the accent label. To jointly train the acoustic model and the accent classifier, we propose the multi-task learning with gate mechanism (MTL-G). As the accent label prediction could be inaccurate, it performs worse than the accent-specific adaptation. Yet, in comparison with the baseline model, MTL-G achieves 5.1% average relative WER reduction.