Listen Like a Teacher: Mitigating Whisper Hallucinations using Adaptive Layer Attention and Knowledge Distillation

TL;DR

This paper introduces a novel two-stage approach combining Adaptive Layer Attention and knowledge distillation to reduce hallucinations in Whisper speech recognition, especially under noisy conditions, improving robustness and accuracy.

Contribution

It proposes a new architecture that enhances Whisper's robustness through adaptive layer attention and knowledge distillation, directly addressing hallucination errors in noisy environments.

Findings

Significant reduction in hallucinations and word error rates in noisy conditions

Improved robustness of Whisper model without sacrificing clean speech performance

Effective use of multi-objective knowledge distillation for noise robustness

Abstract

The Whisper model, an open-source automatic speech recognition system, is widely adopted for its strong performance across multilingual and zero-shot settings. However, it frequently suffers from hallucination errors, especially under noisy acoustic conditions. Previous works to reduce hallucinations in Whisper-style ASR systems have primarily focused on audio preprocessing or post-processing of transcriptions to filter out erroneous content. However, modifications to the Whisper model itself remain largely unexplored to mitigate hallucinations directly. To address this challenge, we present a two-stage architecture that first enhances encoder robustness through Adaptive Layer Attention (ALA) and further suppresses hallucinations using a multi-objective knowledge distillation (KD) framework. In the first stage, ALA groups encoder layers into semantically coherent blocks via inter-layer correlation analysis. A learnable multi-head attention module then fuses these block representations, enabling the model to jointly exploit low- and high-level features for more robust encoding. In the second stage, our KD framework trains the student model on noisy audio to align its semantic and attention distributions with a teacher model processing clean inputs. Our experiments on noisy speech benchmarks show notable reductions in hallucinations and word error rates, while preserving performance on clean speech. Together, ALA and KD offer a principled strategy to improve Whisper's reliability under real-world noisy conditions.