Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels

TL;DR

This paper systematically compares GPT-4's translation performance with human translators across multiple languages, domains, and expertise levels, revealing GPT-4's strengths and limitations in real-world translation tasks.

Contribution

It provides the first comprehensive evaluation of GPT-4 against human translators across languages, domains, and proficiency levels, highlighting its consistent quality and unique translation patterns.

Findings

GPT-4 performs comparably to junior human translators.

GPT-4 maintains consistent quality across resource-poor languages.

Human translators sometimes hallucinate or over-interpret context.

Abstract

This study presents a comprehensive evaluation of GPT-4's translation capabilities compared to human translators of varying expertise levels. Through systematic human evaluation using the MQM schema, we assess translations across three language pairs (Chinese$\longleftrightarrow$English, Russian$\longleftrightarrow$English, and Chinese$\longleftrightarrow$Hindi) and three domains (News, Technology, and Biomedical). Our findings reveal that GPT-4 achieves performance comparable to junior-level translators in terms of total errors, while still lagging behind senior translators. Unlike traditional Neural Machine Translation systems, which show significant performance degradation in resource-poor language directions, GPT-4 maintains consistent translation quality across all evaluated language pairs. Through qualitative analysis, we identify distinctive patterns in translation approaches: GPT-4 tends toward overly literal translations and exhibits lexical inconsistency, while human translators sometimes over-interpret context and introduce hallucinations. This study represents the first systematic comparison between LLM and human translators across different proficiency levels, providing valuable insights into the current capabilities and limitations of LLM-based translation systems.