Benchmarking Machine Translation on Chinese Social Media Texts

TL;DR

This paper introduces CSM-MTBench, a new benchmark for evaluating Chinese social media machine translation, addressing challenges of slang, neologisms, and stylistic fidelity with specialized subsets and evaluation methods.

Contribution

It presents a comprehensive benchmark with tailored evaluation approaches for social media Chinese translation, focusing on slang, style, and informal expressions, filling a critical gap in MT assessment.

Findings

Significant variation in model performance on social media-specific content

Traditional metrics often fail to capture stylistic fidelity

New benchmark enables better assessment of real-world Chinese social media translation

Abstract

The prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese social media, poses significant challenges for Machine Translation (MT) benchmarking. Specifically, we identify two primary obstacles: (1) data scarcity, as high-quality parallel data requires bilingual annotators familiar with platform-specific slang, and stylistic cues in both languages; and (2) metric limitations, where traditional evaluators like COMET often fail to capture stylistic fidelity and nonstandard expressions. To bridge these gaps, we introduce CSM-MTBench, a benchmark covering five Chinese-foreign language directions and consisting of two expert-curated subsets: Fun Posts, featuring context-rich, slang- and neologism-heavy content, and Social Snippets, emphasizing concise, emotion- and style- driven expressions. Furthermore, we propose tailored evaluation approaches for each subset: measuring the translation success rate of slang and neologisms in Fun Posts, while assessing tone and style preservation in Social Snippets via a hybrid of embedding-based metrics and LLM-as-a-judge. Experiments on over 20 models reveal substantial variation in how current MT systems handle semantic fidelity and informal, social-media-specific stylistic cues. CSM-MTBench thus serves as a rigorous testbed for advancing MT systems capable of mastering real-world Chinese social media texts.