Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs

TL;DR

This paper presents the inaugural discourse-level literary translation shared task at WMT 2023, introducing a novel Chinese-English web novel corpus, industry-endorsed evaluation criteria, and analyzing system performance through both automatic and human assessments.

Contribution

It introduces the first discourse-level literary translation shared task, releases a new Chinese-English web novel corpus, and establishes industry-endorsed evaluation standards.

Findings

Multiple systems show varying performance on literary translation tasks.

Human evaluations provide critical insights beyond automatic metrics.

Discourse-aware translation approaches improve literary translation quality.

Abstract

Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the first edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 14 submissions from 7 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at http://www2.statmt.org/wmt23/literary-translation-task.html.