TransBench: Benchmarking Machine Translation for Industrial-Scale Applications

TL;DR

This paper introduces TransBench, a comprehensive benchmark for evaluating industrial machine translation systems, focusing on domain-specific, cultural, and stylistic aspects crucial for real-world applications.

Contribution

It presents a new multi-level evaluation framework, a publicly available benchmark for e-commerce translation, and novel metrics for assessing industry-specific translation quality.

Findings

TransBench covers 33 language pairs and 4 scenarios with 17,000 sentences.

Traditional metrics combined with Marco-MOS improve evaluation accuracy.

Open-source tools facilitate reproducible and systematic assessment.

Abstract

Machine translation (MT) has become indispensable for cross-border communication in globalized industries like e-commerce, finance, and legal services, with recent advancements in large language models (LLMs) significantly enhancing translation quality. However, applying general-purpose MT models to industrial scenarios reveals critical limitations due to domain-specific terminology, cultural nuances, and stylistic conventions absent in generic benchmarks. Existing evaluation frameworks inadequately assess performance in specialized contexts, creating a gap between academic benchmarks and real-world efficacy. To address this, we propose a three-level translation capability framework: (1) Basic Linguistic Competence, (2) Domain-Specific Proficiency, and (3) Cultural Adaptation, emphasizing the need for holistic evaluation across these dimensions. We introduce TransBench, a benchmark tailored for industrial MT, initially targeting international e-commerce with 17,000 professionally translated sentences spanning 4 main scenarios and 33 language pairs. TransBench integrates traditional metrics (BLEU, TER) with Marco-MOS, a domain-specific evaluation model, and provides guidelines for reproducible benchmark construction. Our contributions include: (1) a structured framework for industrial MT evaluation, (2) the first publicly available benchmark for e-commerce translation, (3) novel metrics probing multi-level translation quality, and (4) open-sourced evaluation tools. This work bridges the evaluation gap, enabling researchers and practitioners to systematically assess and enhance MT systems for industry-specific needs.