LFED: A Literary Fiction Evaluation Dataset for Large Language Models

TL;DR

LFED introduces a new dataset for evaluating large language models' understanding of Chinese literary fiction, revealing current models' limited performance and providing insights into factors affecting comprehension.

Contribution

This paper presents LFED, the first comprehensive Chinese literary fiction dataset with a detailed question taxonomy for evaluating LLMs' comprehension and reasoning capabilities.

Findings

LLMs struggle with literary fiction questions, with ChatGPT scoring only 57.08% in zero-shot.

Attributes like novel type and publication year significantly influence LLM performance.

The dataset enables systematic evaluation of LLMs' literary understanding.

Abstract

The rapid evolution of large language models (LLMs) has ushered in the need for comprehensive assessments of their performance across various dimensions. In this paper, we propose LFED, a Literary Fiction Evaluation Dataset, which aims to evaluate the capability of LLMs on the long fiction comprehension and reasoning. We collect 95 literary fictions that are either originally written in Chinese or translated into Chinese, covering a wide range of topics across several centuries. We define a question taxonomy with 8 question categories to guide the creation of 1,304 questions. Additionally, we conduct an in-depth analysis to ascertain how specific attributes of literary fictions (e.g., novel types, character numbers, the year of publication) impact LLM performance in evaluations. Through a series of experiments with various state-of-the-art LLMs, we demonstrate that these models face considerable challenges in effectively addressing questions related to literary fictions, with ChatGPT reaching only 57.08% under the zero-shot setting. The dataset will be publicly available at https://github.com/tjunlp-lab/LFED.git