Instance-Based Model Adaptation For Direct Speech Translation

TL;DR

This paper introduces an instance-based adaptation method for speech translation models that enhances performance by customizing the model on-the-fly using similar data retrieved from a pool, effective across various languages and domains.

Contribution

It proposes a novel instance selection and fine-tuning approach for real-time model adaptation in speech translation, addressing data scarcity issues.

Findings

Performance improvements over static models across multiple scenarios

Effective in low-resource and domain adaptation settings

Applicable to various languages and audio conditions

Abstract

Despite recent technology advancements, the effectiveness of neural approaches to end-to-end speech-to-text translation is still limited by the paucity of publicly available training corpora. We tackle this limitation with a method to improve data exploitation and boost the system's performance at inference time. Our approach allows us to customize "on the fly" an existing model to each incoming translation request. At its core, it exploits an instance selection procedure to retrieve, from a given pool of data, a small set of samples similar to the input query in terms of latent properties of its audio signal. The retrieved samples are then used for an instance-specific fine-tuning of the model. We evaluate our approach in three different scenarios. In all data conditions (different languages, in/out-of-domain adaptation), our instance-based adaptation yields coherent performance gains over static models.