Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression

TL;DR

This paper introduces Orthogonal Softmax within supernet training to efficiently identify optimal subnets for scalable ASR, achieving comparable or better performance than individual models with less resource use.

Contribution

We propose OrthoSoftmax, a novel orthogonal softmax method, enabling efficient, flexible subnet selection within supernet training for scalable automatic speech recognition.

Findings

FLOPs-aware component selection yields best performance.

WERs are comparable or better than individually trained models.

Analysis reveals interesting patterns in selected components.

Abstract

ASR systems are deployed across diverse environments, each with specific hardware constraints. We use supernet training to jointly train multiple encoders of varying sizes, enabling dynamic model size adjustment to fit hardware constraints without redundant training. Moreover, we introduce a novel method called OrthoSoftmax, which applies multiple orthogonal softmax functions to efficiently identify optimal subnets within the supernet, avoiding resource-intensive search. This approach also enables more flexible and precise subnet selection by allowing selection based on various criteria and levels of granularity. Our results with CTC on Librispeech and TED-LIUM-v2 show that FLOPs-aware component-wise selection achieves the best overall performance. With the same number of training updates from one single job, WERs for all model sizes are comparable to or slightly better than those of individually trained models. Furthermore, we analyze patterns in the selected components and reveal interesting insights.