# Diagonal State Space Augmented Transformers for Speech Recognition

## TL;DR

This paper introduces DSS-augmented transformers for speech recognition, replacing convolutions with diagonal state space models, leading to improved WER on multiple datasets and insights into learned basis functions.

## Contribution

The paper proposes a novel DSS-augmented transformer architecture that enhances speech recognition performance over conformers by integrating diagonal state space models.

## Key findings

- Achieved 8.9%/6.7% WER on Switchboard 300/2000 hours.
- Improved WER by 7% on MALACH dataset.
- DSS layers learn damped Fourier basis functions.

## Abstract

We improve on the popular conformer architecture by replacing the depthwise temporal convolutions with diagonal state space (DSS) models. DSS is a recently introduced variant of linear RNNs obtained by discretizing a linear dynamical system with a diagonal state transition matrix. DSS layers project the input sequence onto a space of orthogonal polynomials where the choice of basis functions, metric and support is controlled by the eigenvalues of the transition matrix. We compare neural transducers with either conformer or our proposed DSS-augmented transformer (DSSformer) encoders on three public corpora: Switchboard English conversational telephone speech 300 hours, Switchboard+Fisher 2000 hours, and a spoken archive of holocaust survivor testimonials called MALACH 176 hours. On Switchboard 300/2000 hours, we reach a single model performance of 8.9%/6.7% WER on the combined test set of the Hub5 2000 evaluation, respectively, and on MALACH we improve the WER by 7% relative over the previous best published result. In addition, we present empirical evidence suggesting that DSS layers learn damped Fourier basis functions where the attenuation coefficients are layer specific whereas the frequency coefficients converge to almost identical linearly-spaced values across all layers.