Compute Cost Amortized Transformer for Streaming ASR

TL;DR

This paper introduces a streaming Transformer-based ASR model that dynamically reduces compute costs during inference by creating sparse computation pathways, maintaining high accuracy while significantly lowering resource usage.

Contribution

The paper proposes a novel compute cost amortized Transformer architecture with dynamic sparse computation pathways and an end-to-end training method for efficient streaming ASR.

Findings

Achieves 60% reduction in compute cost

Only 3% relative increase in word error rate

Effective on LibriSpeech dataset

Abstract

We present a streaming, Transformer-based end-to-end automatic speech recognition (ASR) architecture which achieves efficient neural inference through compute cost amortization. Our architecture creates sparse computation pathways dynamically at inference time, resulting in selective use of compute resources throughout decoding, enabling significant reductions in compute with minimal impact on accuracy. The fully differentiable architecture is trained end-to-end with an accompanying lightweight arbitrator mechanism operating at the frame-level to make dynamic decisions on each input while a tunable loss function is used to regularize the overall level of compute against predictive performance. We report empirical results from experiments using the compute amortized Transformer-Transducer (T-T) model conducted on LibriSpeech data. Our best model can achieve a 60% compute cost reduction with only a 3% relative word error rate (WER) increase.