How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs

TL;DR

This paper investigates whether large language models understand the temporal aspects of narratives like humans do, revealing limitations in their narrative comprehension and proposing a new framework for assessment.

Contribution

It introduces a novel probing pipeline and standardized framework to evaluate LLMs' understanding of temporal meaning in narratives, highlighting their differences from human cognition.

Findings

LLMs rely heavily on prototypicality in aspect judgments

They produce inconsistent temporal and aspectual judgments

They struggle with causal reasoning based on narrative aspect

Abstract

Large language models (LLMs) exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these behaviors reflect human-like cognition versus advanced pattern recognition remains an open question. In this study, we investigate how LLMs process the temporal meaning of linguistic aspect in narratives that were previously used in human studies. Using an Expert-in-the-Loop probing pipeline, we conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. Our findings show that LLMs over-rely on prototypicality, produce inconsistent aspectual judgments, and struggle with causal reasoning derived from aspect, raising concerns about their ability to fully comprehend narratives. These results suggest that LLMs process aspect fundamentally differently from humans and lack robust narrative understanding. Beyond these empirical findings, we develop a standardized experimental framework for the reliable assessment of LLMs' cognitive and linguistic capabilities.