Fabricator or dynamic translator?

TL;DR

This paper explores the capabilities and challenges of large language models in machine translation, focusing on their overgeneration behaviors and strategies for detection in commercial applications.

Contribution

It introduces various strategies for identifying and understanding overgeneration in LLM-based translation, with practical insights for commercial deployment.

Findings

LLMs can produce diverse overgeneration types including explanations and confabulations.

Strategies for detecting overgeneration can improve translation reliability.

The work provides practical results from commercial setting experiments.

Abstract

LLMs are proving to be adept at machine translation although due to their generative nature they may at times overgenerate in various ways. These overgenerations are different from the neurobabble seen in NMT and range from LLM self-explanations, to risky confabulations, to appropriate explanations, where the LLM is able to act as a human translator would, enabling greater comprehension for the target audience. Detecting and determining the exact nature of the overgenerations is a challenging task. We detail different strategies we have explored for our work in a commercial setting, and present our results.