Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition

TL;DR

This paper introduces a graph neural network-encoded, tree-constrained pointer generator for end-to-end contextual speech recognition, significantly improving biasing word recognition with minimal additional computational cost.

Contribution

It presents a novel GNN-based encoding method within TCPGen for better biasing word prediction in end-to-end ASR systems.

Findings

Achieved about 15% relative WER reduction on biasing words.

Demonstrated effectiveness on Librispeech and AMI datasets.

Maintained low additional computational cost.

Abstract

Incorporating biasing words obtained as contextual knowledge is critical for many automatic speech recognition (ASR) applications. This paper proposes the use of graph neural network (GNN) encodings in a tree-constrained pointer generator (TCPGen) component for end-to-end contextual ASR. By encoding the biasing words in the prefix-tree with a tree-based GNN, lookahead for future wordpieces in end-to-end ASR decoding is achieved at each tree node by incorporating information about all wordpieces on the tree branches rooted from it, which allows a more accurate prediction of the generation probability of the biasing words. Systems were evaluated on the Librispeech corpus using simulated biasing tasks, and on the AMI corpus by proposing a novel visual-grounded contextual ASR pipeline that extracts biasing words from slides alongside each meeting. Results showed that TCPGen with GNN encodings achieved about a further 15% relative WER reduction on the biasing words compared to the original TCPGen, with a negligible increase in the computation cost for decoding.