HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea

TL;DR

This paper introduces a new dataset and BERT-based models for understanding ancient Hanja documents from Korea, significantly aiding historians by improving language comprehension and analysis of historical texts.

Contribution

The paper releases the Hanja Understanding Evaluation dataset and trains BERT models on historical Korean corpora, demonstrating improved performance on multiple NLP tasks for ancient texts.

Findings

Models trained on historical corpora outperform baselines.

Significant improvements in classification and retrieval tasks.

Zero-shot experiments show potential for analyzing lesser-studied texts.

Abstract

Historical records in Korea before the 20th century were primarily written in Hanja, an extinct language based on Chinese characters and not understood by modern Korean or Chinese speakers. Historians with expertise in this time period have been analyzing the documents, but that process is very difficult and time-consuming, and language models would significantly speed up the process. Toward building and evaluating language models for Hanja, we release the Hanja Understanding Evaluation dataset consisting of chronological attribution, topic classification, named entity recognition, and summary retrieval tasks. We also present BERT-based models continued training on the two major corpora from the 14th to the 19th centuries: the Annals of the Joseon Dynasty and Diaries of the Royal Secretariats. We compare the models with several baselines on all tasks and show there are significant improvements gained by training on the two corpora. Additionally, we run zero-shot experiments on the Daily Records of the Royal Court and Important Officials (DRRI). The DRRI dataset has not been studied much by the historians, and not at all by the NLP community.