Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems

TL;DR

This paper investigates transferring multiple representations from large language models into end-to-end ASR systems, demonstrating that multiple representations can be more effective than a single one for improving speech recognition performance.

Contribution

It introduces techniques for extracting and transferring multiple LLM representations into ASR, showing this approach's effectiveness over single-representation transfer.

Findings

Multiple LLM representations improve ASR accuracy.

Transferring multiple representations is a simple yet effective method.

The approach offers an alternative to single-layer transfer methods.

Abstract

Transferring the knowledge of large language models (LLMs) is a promising technique to incorporate linguistic knowledge into end-to-end automatic speech recognition (ASR) systems. However, existing works only transfer a single representation of LLM (e.g. the last layer of pretrained BERT), while the representation of a text is inherently non-unique and can be obtained variously from different layers, contexts and models. In this work, we explore a wide range of techniques to obtain and transfer multiple representations of LLMs into a transducer-based ASR system. While being conceptually simple, we show that transferring multiple representations of LLMs can be an effective alternative to transferring only a single representation.