Dynamic Acoustic Model Architecture Optimization in Training for ASR

TL;DR

This paper presents DMAO, a novel architecture optimization framework for ASR that automatically reallocates model parameters during training, leading to consistent WER improvements with minimal overhead.

Contribution

DMAO introduces a grow-and-drop strategy for dynamic parameter reallocation during training, enhancing ASR model performance without significant additional computational cost.

Findings

DMAO improves WER by up to 6% relatively across datasets.

The method reallocates parameters from less-utilized to more-beneficial areas.

Analysis reveals patterns in parameter redistribution that inform model design.

Abstract

Architecture design is inherently complex. Existing approaches rely on either handcrafted rules, which demand extensive empirical expertise, or automated methods like neural architecture search, which are computationally intensive. In this paper, we introduce DMAO, an architecture optimization framework that employs a grow-and-drop strategy to automatically reallocate parameters during training. This reallocation shifts resources from less-utilized areas to those parts of the model where they are most beneficial. Notably, DMAO only introduces negligible training overhead at a given model complexity. We evaluate DMAO through experiments with CTC on LibriSpeech, TED-LIUM-v2 and Switchboard datasets. The results show that, using the same amount of training resources, our proposed DMAO consistently improves WER by up to 6% relatively across various architectures, model sizes, and datasets. Furthermore, we analyze the pattern of parameter redistribution and uncover insightful findings.