Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition

TL;DR

This paper investigates the different processing strategies of Transformer and Conformer architectures in speech recognition, revealing that Conformers categorize early while Transformers integrate information later, with implications for their use cases.

Contribution

Introduces Architectural Fingerprinting, a probing framework to analyze and compare the processing hierarchies of Transformer and Conformer speech models.

Findings

Conformers categorize phonemes 29% earlier in depth.

Transformers defer phoneme and speaker encoding to deeper layers.

Different hierarchies suggest distinct advantages for low-latency and context-rich tasks.

Abstract

In speech language modeling, two architectures dominate the frontier: the Transformer and the Conformer. However, it remains unknown whether their comparable performance stems from convergent processing strategies or distinct architectural inductive biases. We introduce Architectural Fingerprinting, a probing framework that isolates the effect of architecture on representation, and apply it to a controlled suite of 24 pre-trained encoders (39M-3.3B parameters). Our analysis reveals divergent hierarchies: Conformers implement a "Categorize Early" strategy, resolving phoneme categories 29% earlier in depth and speaker gender by 16% depth. In contrast, Transformers "Integrate Late," deferring phoneme, accent, and duration encoding to deep layers (49-57%). These fingerprints suggest design heuristics: Conformers' front-loaded categorization may benefit low-latency streaming, while Transformers' deep integration may favor tasks requiring rich context and cross-utterance normalization.