Advancing Explainability in Neural Machine Translation: Analytical Metrics for Attention and Alignment Consistency

TL;DR

This paper introduces quantitative metrics to evaluate the explainability of neural machine translation models by analyzing attention patterns and their correlation with translation quality, aiming to improve transparency.

Contribution

It proposes a systematic framework with new metrics for assessing attention interpretability and validates them on a standard dataset, enhancing understanding of NMT explainability.

Findings

Sharper attention distributions correlate with better interpretability.

Attention quality does not always align with translation performance.

The framework aids in developing more transparent NMT systems.

Abstract

Neural Machine Translation (NMT) models have shown remarkable performance but remain largely opaque in their decision making processes. The interpretability of these models, especially their internal attention mechanisms, is critical for building trust and verifying that these systems behave as intended. In this work, we introduce a systematic framework to quantitatively evaluate the explainability of an NMT model attention patterns by comparing them against statistical alignments and correlating them with standard machine translation quality metrics. We present a set of metrics attention entropy and alignment agreement and validate them on an English-German test subset from WMT14 using a pre trained mT5 model. Our results indicate that sharper attention distributions correlate with improved interpretability but do not always guarantee better translation quality. These findings advance our understanding of NMT explainability and guide future efforts toward building more transparent and reliable machine translation systems.