Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry

TL;DR

This study explores how large language models understand ancient Chinese poetry by collecting poems, designing comprehension metrics, and analyzing literary patterns to support future AI-driven literary creation.

Contribution

It introduces a novel methodology for evaluating literary understanding of LLMs using ancient Chinese poetry and analyzes literary patterns through correlation analysis.

Findings

LLMs can effectively evaluate ancient Chinese poems.

Identified correlations between poem collections and literary patterns.

Provided insights for improving AI-based literary creation.

Abstract

The birth and rapid development of large language models (LLMs) have caused quite a stir in the field of literature. Once considered unattainable, AI's role in literary creation is increasingly becoming a reality. In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. However, it's difficult to further improve the quality of these works. This is primarily because understanding and appreciating a good literary work involves a considerable threshold, such as knowledge of literary theory, aesthetic sensibility, interdisciplinary knowledge. Therefore, authoritative data in this area is quite lacking. Additionally, evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. To address this issue, this paper attempts to explore the mysteries of literary texts from the perspective of LLMs, using ancient Chinese poetry as an example for experimentation. First, we collected a variety of ancient poems from different sources and had experts annotate a small portion of them. Then, we designed a range of comprehension metrics based on LLMs to evaluate all these poems. Finally, we analyzed the correlations and differences between various poem collections to identify literary patterns. Through our experiments, we observed a series of enlightening phenomena that provide technical support for the future development of high-level literary creation based on LLMs.