Attention-Free Keyword Spotting

TL;DR

This paper investigates replacing attention mechanisms with gated MLPs for keyword spotting, demonstrating that MLP-based models can achieve competitive accuracy with significantly fewer parameters.

Contribution

It introduces a family of efficient MLP-based models for keyword spotting that outperform attention-based models in parameter efficiency.

Findings

MLP-based models achieve competitive accuracy on Google Speech Commands benchmarks.

Models have less than 0.5 million parameters, significantly fewer than attention-based counterparts.

The approach demonstrates the viability of attention-free models for speech recognition tasks.

Abstract

Till now, attention-based models have been used with great success in the keyword spotting problem domain. However, in light of recent advances in deep learning, the question arises whether self-attention is truly irreplaceable for recognizing speech keywords. We thus explore the usage of gated MLPs --previously shown to be alternatives to transformers in vision tasks-- for the keyword spotting task. We provide a family of highly efficient MLP-based models for keyword spotting, with less than 0.5 million parameters. We show that our approach achieves competitive performance on Google Speech Commands V2-12 and V2-35 benchmarks with much fewer parameters than self-attention-based methods.