GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning

TL;DR

GrammaMT introduces a grammar-aware prompting method for machine translation that leverages linguistic annotations, significantly improving translation quality especially in low-resource scenarios without additional training.

Contribution

It proposes three training-free prompting strategies utilizing Interlinear Glossed Text to enhance machine translation performance across diverse languages and datasets.

Findings

Improves translation BLEU scores by over 17 points with gloss resources.

Effective across low-resource, endangered, and out-of-domain language datasets.

Demonstrates minimal effort collection of linguistic annotations can boost translation quality.

Abstract

We introduce GrammaMT, a grammatically-aware prompting approach for machine translation that uses Interlinear Glossed Text (IGT), a common form of linguistic description providing morphological and lexical annotations for source sentences. GrammaMT proposes three prompting strategies: gloss-shot, chain-gloss and model-gloss. All are training-free, requiring only a few examples that involve minimal effort to collect, and making them well-suited for low-resource setups. Experiments show that GrammaMT enhances translation performance on open-source instruction-tuned LLMs for various low- to high-resource languages across three benchmarks: (1) the largest IGT corpus, (2) the challenging 2023 SIGMORPHON Shared Task data over endangered languages, and (3) even in an out-of-domain setting with FLORES. Moreover, ablation studies reveal that leveraging gloss resources could substantially boost MT performance (by over 17 BLEU points) if LLMs accurately generate or access input sentence glosses.