Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation

TL;DR

This paper introduces Viram, a benchmark for testing punctuation robustness in English-Marathi NMT, and evaluates strategies to improve translation quality when punctuation is missing or incorrect.

Contribution

The work presents a new benchmark dataset and compares remediation strategies, showing their effectiveness over existing models in handling punctuation errors.

Findings

Both remediation strategies improve NMT performance significantly.

Current LLMs are less robust than task-specific strategies for punctuation errors.

Viram benchmark exposes weaknesses in existing NMT systems regarding punctuation robustness.

Abstract

Neural Machine Translation (NMT) systems rely heavily on explicit punctuation cues to resolve semantic ambiguities in a source sentence. Inputting user-generated sentences, which are likely to contain missing or incorrect punctuation, results in fluent but semantically disastrous translations. This work attempts to highlight and address the problem of punctuation robustness of NMT systems through an English-to-Marathi translation. First, we introduce \textbf{\textit{Viram}}, a human-curated diagnostic benchmark of 54 punctuation-ambiguous English-Marathi sentence pairs to stress-test existing NMT systems. Second, we evaluate two simple remediation strategies: cascade-based \textit{restore-then-translate} and \textit{direct fine-tuning}. Our experimental results and analysis demonstrate that both strategies yield substantial NMT performance improvements. Furthermore, we find that current Large Language Models (LLMs) exhibit relatively poorer robustness in translating such sentences than these task-specific strategies, thus necessitating further research in this area. The code and dataset are available at https://github.com/KaustubhShejole/Viram_Marathi.