Named Entity Detection and Injection for Direct Speech Translation

TL;DR

This paper introduces a method to improve speech-to-text translation by detecting and injecting named entities using dictionaries, significantly reducing errors in person names and locations.

Contribution

It presents a novel approach to detect and incorporate known named entities into S2T translation, enhancing accuracy for critical words.

Findings

31% reduction in person name errors

Reliable detection of named entities from encoder outputs

Improved translation quality for location and person names

Abstract

In a sentence, certain words are critical for its semantic. Among them, named entities (NEs) are notoriously challenging for neural models. Despite their importance, their accurate handling has been neglected in speech-to-text (S2T) translation research, and recent work has shown that S2T models perform poorly for locations and notably person names, whose spelling is challenging unless known in advance. In this work, we explore how to leverage dictionaries of NEs known to likely appear in a given context to improve S2T model outputs. Our experiments show that we can reliably detect NEs likely present in an utterance starting from S2T encoder outputs. Indeed, we demonstrate that the current detection quality is sufficient to improve NE accuracy in the translation with a 31% reduction in person name errors.