USM RNN-T model weights binarization

TL;DR

This paper introduces a weights binarization method for USM RNN-T models, significantly reducing model size by 15.9 times with minimal impact on accuracy, making large-scale speech models more practical for deployment.

Contribution

The paper presents a novel weights binarization technique for USM RNN-T models, achieving substantial size reduction with minimal WER increase.

Findings

Model size reduced by 15.9x

WER increased by only 1.9%

Binarization is practical for deployment

Abstract

Large-scale universal speech models (USM) are already used in production. However, as the model size grows, the serving cost grows too. Serving cost of large models is dominated by model size that is why model size reduction is an important research topic. In this work we are focused on model size reduction using weights only quantization. We present the weights binarization of USM Recurrent Neural Network Transducer (RNN-T) and show that its model size can be reduced by 15.9x times at cost of word error rate (WER) increase by only 1.9% in comparison to the float32 model. It makes it attractive for practical applications.