KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning

TL;DR

KRISTEVA introduces a novel benchmark for assessing large language models' interpretive reasoning in close reading, a key literary analysis skill, revealing current models' limitations compared to human experts.

Contribution

This paper presents the first large-scale close reading benchmark, KRISTEVA, with tasks designed to evaluate interpretive reasoning in literary analysis for LLMs.

Findings

LLMs achieve 49.7%-69.7% accuracy on close reading tasks

Models underperform compared to human evaluators on most tasks

KRISTEVA enables systematic evaluation of interpretive reasoning in LLMs

Abstract

Each year, tens of millions of essays are written and graded in college-level English courses. Students are asked to analyze literary and cultural texts through a process known as close reading, in which they gather textual details to formulate evidence-based arguments. Despite being viewed as a basis for critical thinking and widely adopted as a required element of university coursework, close reading has never been evaluated on large language models (LLMs), and multi-discipline benchmarks like MMLU do not include literature as a subject. To fill this gap, we present KRISTEVA, the first close reading benchmark for evaluating interpretive reasoning, consisting of 1331 multiple-choice questions adapted from classroom data. With KRISTEVA, we propose three progressively more difficult sets of tasks to approximate different elements of the close reading process, which we use to test how well LLMs may seem to understand and reason about literary works: 1) extracting stylistic features, 2) retrieving relevant contextual information from parametric knowledge, and 3) multi-hop reasoning between style and external contexts. Our baseline results find that, while state-of-the-art LLMs possess some college-level close reading competency (accuracy 49.7% - 69.7%), their performances still trail those of experienced human evaluators on 10 out of our 11 tasks.