Learned Transferable Architectures Can Surpass Hand-Designed Architectures for Large Scale Speech Recognition

TL;DR

This paper demonstrates that neural architecture search (NAS) can find transferable speech recognition architectures that outperform hand-designed models on large datasets, reducing computational costs and enabling industrial-scale applications.

Contribution

The study introduces a revised search space for NAS in speech recognition and shows that architectures found on small datasets transfer effectively to large-scale datasets, outperforming hand-designed models.

Findings

Transferred architectures outperform hand-designed models on large datasets.

Revised search space reduces computational overhead and memory usage.

NAS architectures achieve over 20% relative improvement on AISHELL-2.

Abstract

In this paper, we explore the neural architecture search (NAS) for automatic speech recognition (ASR) systems. With reference to the previous works in the computer vision field, the transferability of the searched architecture is the main focus of our work. The architecture search is conducted on the small proxy dataset, and then the evaluation network, constructed with the searched architecture, is evaluated on the large dataset. Especially, we propose a revised search space for speech recognition tasks which theoretically facilitates the search algorithm to explore the architectures with low complexity. Extensive experiments show that: (i) the architecture searched on the small proxy dataset can be transferred to the large dataset for the speech recognition tasks. (ii) the architecture learned in the revised search space can greatly reduce the computational overhead and GPU memory usage with mild performance degradation. (iii) the searched architecture can achieve more than 20% and 15% (average on the four test sets) relative improvements respectively on the AISHELL-2 dataset and the large (10k hours) dataset, compared with our best hand-designed DFSMN-SAN architecture. To the best of our knowledge, this is the first report of NAS results with large scale dataset (up to 10K hours), indicating the promising application of NAS to industrial ASR systems.