Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?

TL;DR

This paper investigates whether large language models can estimate the cognitive complexity of reading comprehension items, focusing on evidence scope and transformation level, to aid in pre-assessment difficulty analysis.

Contribution

It demonstrates that LLMs can approximate cognitive complexity of RC items, highlighting their potential for automated prior difficulty estimation.

Findings

LLMs can estimate cognitive complexity dimensions of RC items.

A gap exists between LLM reasoning ability and metacognitive awareness.

LLMs show promise as tools for pre-assessment difficulty analysis.

Abstract

Estimating the cognitive complexity of reading comprehension (RC) items is crucial for assessing item difficulty before it is administered to learners. Unlike syntactic and semantic features, such as passage length or semantic similarity between options, cognitive features that arise during answer reasoning are not readily extractable using existing NLP tools and have traditionally relied on human annotation. In this study, we examine whether large language models (LLMs) can estimate the cognitive complexity of RC items by focusing on two dimensions-Evidence Scope and Transformation Level-that indicate the degree of cognitive burden involved in reasoning about the answer. Our experimental results demonstrate that LLMs can approximate the cognitive complexity of items, indicating their potential as tools for prior difficulty analysis. Further analysis reveals a gap between LLMs' reasoning ability and their metacognitive awareness: even when they produce correct answers, they sometimes fail to correctly identify the features underlying their own reasoning process.