An Empirical Study of In-context Learning in LLMs for Machine Translation

TL;DR

This paper provides an extensive empirical analysis of in-context learning in large language models for machine translation, revealing key factors influencing performance and the surprising effectiveness of cross-task demonstrations.

Contribution

It is the first comprehensive study examining how various aspects of examples affect in-context learning for machine translation in LLMs.

Findings

ICL is primarily example-driven, not instruction-driven.

Quality of target distribution impacts performance more than source.

Perturbations can act as regularizers, improving results.

Abstract

Recent interest has surged in employing Large Language Models (LLMs) for machine translation (MT) via in-context learning (ICL) (Vilar et al., 2023). Most prior studies primarily focus on optimizing translation quality, with limited attention to understanding the specific aspects of ICL that influence the said quality. To this end, we perform the first of its kind, an exhaustive study of in-context learning for machine translation. We first establish that ICL is primarily example-driven and not instruction-driven. Following this, we conduct an extensive exploration of various aspects of the examples to understand their influence on downstream performance. Our analysis includes factors such as quality and quantity of demonstrations, spatial proximity, and source versus target originality. Further, we also investigate challenging scenarios involving indirectness and misalignment of examples to understand the limits of ICL. While we establish the significance of the quality of the target distribution over the source distribution of demonstrations, we further observe that perturbations sometimes act as regularizers, resulting in performance improvements. Surprisingly, ICL does not necessitate examples from the same task, and a related task with the same target distribution proves sufficient. We hope that our study acts as a guiding resource for considerations in utilizing ICL for MT. Our code is available on https://github.com/PranjalChitale/in-context-mt-analysis.