InteChar: A Unified Oracle Bone Character List for Ancient Chinese Language Modeling

TL;DR

This paper introduces InteChar, a comprehensive character list for ancient Chinese scripts, enabling better digitization and modeling of historical texts, and demonstrates its effectiveness through experiments on an expert-annotated corpus.

Contribution

We propose InteChar, a unified character list for ancient Chinese scripts, and create OracleCS, a new corpus, to improve language modeling of historical texts.

Findings

Models with InteChar outperform baselines on historical language tasks.

InteChar enables consistent digitization of ancient Chinese texts.

The approach facilitates future research in ancient Chinese NLP.

Abstract

Constructing historical language models (LMs) plays a crucial role in aiding archaeological provenance studies and understanding ancient cultures. However, existing resources present major challenges for training effective LMs on historical texts. First, the scarcity of historical language samples renders unsupervised learning approaches based on large text corpora highly inefficient, hindering effective pre-training. Moreover, due to the considerable temporal gap and complex evolution of ancient scripts, the absence of comprehensive character encoding schemes limits the digitization and computational processing of ancient texts, particularly in early Chinese writing. To address these challenges, we introduce InteChar, a unified and extensible character list that integrates unencoded oracle bone characters with traditional and modern Chinese. InteChar enables consistent digitization and representation of historical texts, providing a foundation for robust modeling of ancient scripts. To evaluate the effectiveness of InteChar, we construct the Oracle Corpus Set (OracleCS), an ancient Chinese corpus that combines expert-annotated samples with LLM-assisted data augmentation, centered on Chinese oracle bone inscriptions. Extensive experiments show that models trained with InteChar on OracleCS achieve substantial improvements across various historical language understanding tasks, confirming the effectiveness of our approach and establishing a solid foundation for future research in ancient Chinese NLP.