# Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation

## TL;DR

This paper introduces an elitist sampling method combined with ensemble knowledge distillation to improve domain generalization in automatic speech recognition, outperforming traditional averaging techniques.

## Contribution

It proposes a novel elitist sampling strategy for ensemble teacher models to enhance domain adaptation in speech recognition systems.

## Key findings

- Student model achieves at least 8.4% WER improvement over teachers and baselines.
- Sampling based on individual model posteriors improves adaptation to unseen domains.
- Correlation analysis provides insights into model adaptation with out-of-domain data.

## Abstract

Knowledge distillation has widely been used for model compression and domain adaptation for speech applications. In the presence of multiple teachers, knowledge can easily be transferred to the student by averaging the models output. However, previous research shows that the student do not adapt well with such combination. This paper propose to use an elitist sampling strategy at the output of ensemble teacher models to select the best-decoded utterance generated by completely out-of-domain teacher models for generalizing unseen domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the student is adapted for the Switchboard data. The results show that with the selection strategy based on the individual models posteriors the student model achieves a better WER compared to all the teachers and baselines with a minimum absolute improvement of about 8.4 percent. Furthermore, an insights on the model adaptation with out-of-domain data has also been studied via correlation analysis.