Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systems

TL;DR

Translation Canvas is an explainable interface that helps researchers analyze and understand machine translation system performance at both system and instance levels, addressing limitations of existing evaluation tools.

Contribution

It introduces a novel explainable interface for detailed translation system analysis, improving interpretability over traditional single-score metrics.

Findings

Human evaluation shows superior enjoyability and understandability.

Effectively identifies common errors and analyzes system relationships.

Supports fine-grained error span explanations.

Abstract

With the rapid advancement of machine translation research, evaluation toolkits have become essential for benchmarking system progress. Tools like COMET and SacreBLEU offer single quality score assessments that are effective for pairwise system comparisons. However, these tools provide limited insights for fine-grained system-level comparisons and the analysis of instance-level defects. To address these limitations, we introduce Translation Canvas, an explainable interface designed to pinpoint and analyze translation systems' performance: 1) Translation Canvas assists machine translation researchers in comprehending system-level model performance by identifying common errors (their frequency and severity) and analyzing relationships between different systems based on various evaluation metrics. 2) It supports fine-grained analysis by highlighting error spans with explanations and selectively displaying systems' predictions. According to human evaluation, Translation Canvas demonstrates superior performance over COMET and SacreBLEU packages under enjoyability and understandability criteria.