Analyzing Context Contributions in LLM-based Machine Translation

TL;DR

This paper investigates how large language models utilize different parts of input context in machine translation, revealing key patterns and biases that influence translation quality and can help identify errors.

Contribution

It provides a detailed analysis of context contribution patterns in LLM-based MT, highlighting effects of finetuning and positional biases, and proposes using contribution inspection to detect translation issues.

Findings

Source context contributes more than target context in few-shot examples.

Finetuning changes how different context parts are used.

Earlier examples have higher influence on translation output.

Abstract

Large language models (LLMs) have achieved state-of-the-art performance in machine translation (MT) and demonstrated the ability to leverage in-context learning through few-shot examples. However, the mechanisms by which LLMs use different parts of the input context remain largely unexplored. In this work, we provide a comprehensive analysis of context utilization in MT, studying how LLMs use various context parts, such as few-shot examples and the source text, when generating translations. We highlight several key findings: (1) the source part of few-shot examples appears to contribute more than its corresponding targets, irrespective of translation direction; (2) finetuning LLMs with parallel data alters the contribution patterns of different context parts; and (3) there is a positional bias where earlier few-shot examples have higher contributions to the translated sequence. Finally, we demonstrate that inspecting anomalous context contributions can potentially uncover pathological translations, such as hallucinations. Our findings shed light on the internal workings of LLM-based MT which go beyond those known for standard encoder-decoder MT models.