Activation Steering for Accent Adaptation in Speech Foundation Models

TL;DR

This paper introduces a novel, interpretable method for accent adaptation in speech recognition by identifying and controlling accent-related variations in model activations, leading to improved accuracy without model fine-tuning.

Contribution

It proposes a layer-wise accent sensitivity analysis and a parameter-free activation steering technique for effective accent adaptation in speech models.

Findings

Accent information is concentrated in middle encoder layers.

Activation steering reduces word error rates across multiple accents.

The method does not require model fine-tuning or additional data.

Abstract

Accent variability remains a major errors in automatic speech recognition, yet most adaptation methods rely on parameter fine-tuning without understanding where accent information is encoded. We treat accent variation as an interpretable subspace in hidden representations and investigate whether it can be identified and controlled directly in activation space. We extract layer-wise encoder activations and estimate mean-shift directions capturing accent-induced representation shifts. By injecting these directions into individual layers and measuring how they align accented and standard embeddings, we derive a layer-wise accent sensitivity profile, revealing that accent information concentrates in a narrow band of middle encoder layers. Leveraging this structure, we further introduce parameter-free accent steering that modifies representations during inference without updating model weights. Experiments across eight accents show consistent word error rate reductions.