Individualized Cognitive Simulation in Large Language Models: Evaluating Different Cognitive Representation Methods

TL;DR

This paper evaluates methods for enabling large language models to simulate individual human thought processes, focusing on authorial style and cognitive representation techniques to improve personalized AI storytelling.

Contribution

It introduces a novel benchmark and evaluates various cognitive representation methods, revealing that combining conceptual and linguistic features enhances individualized cognitive simulation in LLMs.

Findings

Combining conceptual and linguistic features improves simulation accuracy.

LLMs better mimic linguistic style than narrative structure.

Effective cognitive representations aid in personalized storytelling.

Abstract

Individualized cognitive simulation (ICS) aims to build computational models that approximate the thought processes of specific individuals. While large language models (LLMs) convincingly mimic surface-level human behavior such as role-play, their ability to simulate deeper individualized cognitive processes remains poorly understood. To address this gap, we introduce a novel task that evaluates different cognitive representation methods in ICS. We construct a dataset from recently published novels (later than the release date of the tested LLMs) and propose an 11-condition cognitive evaluation framework to benchmark seven off-the-shelf LLMs in the context of authorial style emulation. We hypothesize that effective cognitive representations can help LLMs generate storytelling that better mirrors the original author. Thus, we test different cognitive representations, e.g., linguistic features, concept mappings, and profile-based information. Results show that combining conceptual and linguistic features is particularly effective in ICS, outperforming static profile-based cues in overall evaluation. Importantly, LLMs are more effective at mimicking linguistic style than narrative structure, underscoring their limits in deeper cognitive simulation. These findings provide a foundation for developing AI systems that adapt to individual ways of thinking and expression, advancing more personalized and human-aligned creative technologies.