Knowledge Distillation for Improved Accuracy in Spoken Question Answering

TL;DR

This paper introduces a knowledge distillation framework that enhances spoken question answering accuracy by leveraging both spoken and written documents, effectively reducing transcription noise impact.

Contribution

It proposes a novel distillation training strategy from spoken and written texts, improving model performance on noisy ASR transcripts in SQA tasks.

Findings

Outperforms state-of-the-art language models on Spoken-SQuAD

Reduces the impact of transcription noise on SQA accuracy

Improves model robustness with knowledge distillation

Abstract

Spoken question answering (SQA) is a challenging task that requires the machine to fully understand the complex spoken documents. Automatic speech recognition (ASR) plays a significant role in the development of QA systems. However, the recent work shows that ASR systems generate highly noisy transcripts, which critically limit the capability of machine comprehension on the SQA task. To address the issue, we present a novel distillation framework. Specifically, we devise a training strategy to perform knowledge distillation (KD) from spoken documents and written counterparts. Our work makes a step towards distilling knowledge from the language model as a supervision signal to lead to better student accuracy by reducing the misalignment between automatic and manual transcriptions. Experiments demonstrate that our approach outperforms several state-of-the-art language models on the Spoken-SQuAD dataset.