Insights on Neural Representations for End-to-End Speech Recognition

TL;DR

This paper investigates the internal layer-wise neural representations of end-to-end speech recognition models using correlation analysis techniques, revealing architecture-specific dynamics that can inform the design of improved models.

Contribution

It introduces the use of CCA and CKA to analyze neural representations in CNN, LSTM, and Transformer ASR models, providing new insights into their internal dynamics during training.

Findings

CNN layers show hierarchical correlation dependencies

LSTM layers exhibit bottom-up correlation patterns

Transformers display irregular correlation patterns

Abstract

End-to-end automatic speech recognition (ASR) models aim to learn a generalised speech representation. However, there are limited tools available to understand the internal functions and the effect of hierarchical dependencies within the model architecture. It is crucial to understand the correlations between the layer-wise representations, to derive insights on the relationship between neural representations and performance. Previous investigations of network similarities using correlation analysis techniques have not been explored for End-to-End ASR models. This paper analyses and explores the internal dynamics between layers during training with CNN, LSTM and Transformer based approaches using Canonical correlation analysis (CCA) and centered kernel alignment (CKA) for the experiments. It was found that neural representations within CNN layers exhibit hierarchical correlation dependencies as layer depth increases but this is mostly limited to cases where neural representation correlates more closely. This behaviour is not observed in LSTM architecture, however there is a bottom-up pattern observed across the training process, while Transformer encoder layers exhibit irregular coefficiency correlation as neural depth increases. Altogether, these results provide new insights into the role that neural architectures have upon speech recognition performance. More specifically, these techniques can be used as indicators to build better performing speech recognition models.