Learning ASR-Robust Contextualized Embeddings for Spoken Language Understanding

TL;DR

This paper introduces a confusion-aware fine-tuning approach to enhance the robustness of contextualized embeddings from pre-trained language models against ASR errors, significantly improving spoken language understanding performance.

Contribution

It presents a novel fine-tuning method that makes language models more resilient to ASR errors by aligning representations of acoustically confusable words.

Findings

Significant performance improvement on ATIS dataset with ASR transcripts.

Effective mitigation of ASR errors in spoken language understanding.

Source code publicly available for reproducibility.

Abstract

Employing pre-trained language models (LM) to extract contextualized word representations has achieved state-of-the-art performance on various NLP tasks. However, applying this technique to noisy transcripts generated by automatic speech recognizer (ASR) is concerned. Therefore, this paper focuses on making contextualized representations more ASR-robust. We propose a novel confusion-aware fine-tuning method to mitigate the impact of ASR errors to pre-trained LMs. Specifically, we fine-tune LMs to produce similar representations for acoustically confusable words that are obtained from word confusion networks (WCNs) produced by ASR. Experiments on the benchmark ATIS dataset show that the proposed method significantly improves the performance of spoken language understanding when performing on ASR transcripts. Our source code is available at https://github.com/MiuLab/SpokenVec