Learnable Pulse Accumulation for On-Device Speech Recognition: How Much Attention Do You Need?

TL;DR

This paper introduces the Learnable Pulse Accumulator (LPA), a novel O(n) attention mechanism that significantly reduces computational complexity in transformer-based speech models, enabling efficient on-device speech recognition with minimal accuracy loss.

Contribution

The paper presents LPA, a learnable gating mechanism replacing quadratic attention with linear complexity, optimized for edge devices, and demonstrates its effectiveness on speech recognition and enhancement tasks.

Findings

Replacing 8 of 12 layers yields 10.61% WER on LibriSpeech test-clean.

LPA achieves 3.27x speedup on Apple M4 Pro with minimal accuracy loss.

All intra-chunk attention layers can be replaced without collapse in speech enhancement.

Abstract

Self-attention scales quadratically with sequence length, limiting transformer-based speech models on edge devices. We introduce the Learnable Pulse Accumulator (LPA), an O(n) replacement that substitutes key-query dot products with learned gating functions: content-dependent rectangular pulses, periodic windows, and position-dependent basis functions. An MSE diagnostic sweep determines per-layer replacement difficulty and ordering. Replacing 8 of 12 wav2vec2-base layers yields 10.61% word error rate (WER) on LibriSpeech test-clean, +7.24 percentage points (pp) over the 3.37% baseline, with 3.27x speedup at 120s audio on Apple M4 Pro via an optimized MLX inference path. Cross-domain validation on SepFormer speech enhancement shows all 16 intra-chunk attention layers can be replaced without collapse, suggesting the depth wall arises from linguistic computation rather than an LPA limitation. LPA's near-binary gates at inference enable dense GPU computation with no CPU-GPU synchronization, and all operations map to mobile neural accelerators.