AncientBench: Towards Comprehensive Evaluation on Excavated and Transmitted Chinese Corpora

TL;DR

AncientBench is a comprehensive benchmark designed to evaluate large language models' understanding of ancient Chinese texts, especially excavated documents, across multiple comprehension dimensions.

Contribution

The paper introduces AncientBench, a new benchmark with ten tasks for assessing ancient Chinese text comprehension, including excavated documents, and provides baseline evaluations with large language models.

Findings

Large language models show potential in ancient Chinese text comprehension.

Significant gap exists between model performance and human understanding.

AncientBench covers four key competencies of ancient character comprehension.

Abstract

Comprehension of ancient texts plays an important role in archaeology and understanding of Chinese history and civilization. The rapid development of large language models needs benchmarks that can evaluate their comprehension of ancient characters. Existing Chinese benchmarks are mostly targeted at modern Chinese and transmitted documents in ancient Chinese, but the part of excavated documents in ancient Chinese is not covered. To meet this need, we propose the AncientBench, which aims to evaluate the comprehension of ancient characters, especially in the scenario of excavated documents. The AncientBench is divided into four dimensions, which correspond to the four competencies of ancient character comprehension: glyph comprehension, pronunciation comprehension, meaning comprehension, and contextual comprehension. The benchmark also contains ten tasks, including radical, phonetic radical, homophone, cloze, translation, and more, providing a comprehensive framework for evaluation. We convened archaeological researchers to conduct experimental evaluations, proposed an ancient model as baseline, and conducted extensive experiments on the currently best-performing large language models. The experimental results reveal the great potential of large language models in ancient textual scenarios as well as the gap with humans. Our research aims to promote the development and application of large language models in the field of archaeology and ancient Chinese language.