Towards Effective Ancient Chinese Translation: Dataset, Model, and Evaluation

TL;DR

This paper introduces Erya, a new dataset, model, and evaluation benchmark for ancient Chinese translation, demonstrating superior zero-shot and fine-tuned performance over existing models.

Contribution

The paper presents a comprehensive ancient Chinese dataset, a novel training method with two tasks, and a benchmark for evaluating translation quality, advancing the field significantly.

Findings

Erya achieves +12.0 BLEU over GPT-3.5 in zero-shot scenarios.

Erya outperforms ERNIE Bot in human evaluations.

Fine-tuning Erya yields +6.2 BLEU gain.

Abstract

Interpreting ancient Chinese has been the key to comprehending vast Chinese literature, tradition, and civilization. In this paper, we propose Erya for ancient Chinese translation. From a dataset perspective, we collect, clean, and classify ancient Chinese materials from various sources, forming the most extensive ancient Chinese resource to date. From a model perspective, we devise Erya training method oriented towards ancient Chinese. We design two jointly-working tasks: disyllabic aligned substitution (DAS) and dual masked language model (DMLM). From an evaluation perspective, we build a benchmark to judge ancient Chinese translation quality in different scenarios and evaluate the ancient Chinese translation capacities of various existing models. Our model exhibits remarkable zero-shot performance across five domains, with over +12.0 BLEU against GPT-3.5 models and better human evaluation results than ERNIE Bot. Subsequent fine-tuning further shows the superior transfer capability of Erya model with +6.2 BLEU gain. We release all the above-mentioned resources at https://github.com/RUCAIBox/Erya.