Similarity and Content-based Phonetic Self Attention for Speech Recognition

TL;DR

This paper introduces a novel phonetic self-attention mechanism combining similarity and content-based attention to enhance speech recognition by capturing more representative phonetic features without increasing latency or model size.

Contribution

The paper proposes a new phonetic self-attention method that improves speech recognition by explicitly modeling phonetic features through dual attention types, a novel approach compared to standard self-attention.

Findings

Improved phoneme classification accuracy.

Enhanced speech recognition performance.

No increase in latency or parameter size.

Abstract

Transformer-based speech recognition models have achieved great success due to the self-attention (SA) mechanism that utilizes every frame in the feature extraction process. Especially, SA heads in lower layers capture various phonetic characteristics by the query-key dot product, which is designed to compute the pairwise relationship between frames. In this paper, we propose a variant of SA to extract more representative phonetic features. The proposed phonetic self-attention (phSA) is composed of two different types of phonetic attention; one is similarity-based and the other is content-based. In short, similarity-based attention captures the correlation between frames while content-based attention only considers each frame without being affected by other frames. We identify which parts of the original dot product equation are related to two different attention patterns and improve each part with simple modifications. Our experiments on phoneme classification and speech recognition show that replacing SA with phSA for lower layers improves the recognition performance without increasing the latency and the parameter size.