Do we really need Self-Attention for Streaming Automatic Speech Recognition?

TL;DR

This paper critically evaluates the necessity of self-attention in streaming automatic speech recognition, demonstrating that alternative methods like deformable convolution can reduce computational costs without significant performance loss.

Contribution

The study shows that self-attention can be replaced or removed in streaming ASR models, offering more efficient alternatives without degrading accuracy.

Findings

Deformable convolution reduces computational cost in streaming ASR.

Removing self-attention does not significantly impact Word Error Rate.

Self-attention is not essential for effective streaming ASR.

Abstract

Transformer-based architectures are the most used architectures in many deep learning fields like Natural Language Processing, Computer Vision or Speech processing. It may encourage the direct use of Transformers in the constrained tasks, without questioning whether it will yield the same benefits as in standard tasks. Given specific constraints, it is essential to evaluate the relevance of transformer models. This work questions the suitability of transformers for specific domains. We argue that the high computational requirements and latency issues associated with these models do not align well with streaming applications. Our study promotes the search for alternative strategies to improve efficiency without sacrificing performance. In light of this observation, our paper critically examines the usefulness of transformer architecture in such constrained environments. As a first attempt, we show that the computational cost for Streaming Automatic Speech Recognition (ASR) can be reduced using deformable convolution instead of Self-Attention. Furthermore, we show that Self-Attention mechanisms can be entirely removed and not replaced, without observing significant degradation in the Word Error Rate.