InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Styles

TL;DR

InMind is a new evaluation framework that assesses whether large language models can understand and adapt to individual human reasoning styles in social deduction games, revealing current models' limitations in personalized reasoning.

Contribution

The paper introduces InMind, a cognitively grounded framework for evaluating LLMs' ability to capture and apply personalized reasoning styles in social deduction contexts.

Findings

GPT-4o relies on lexical cues and struggles with temporal reasoning.

Reasoning-enhanced models like DeepSeek-R1 show early signs of style-sensitive reasoning.

Current LLMs have limited capacity for individualized, adaptive reasoning.

Abstract

LLMs have shown strong performance on human-centric reasoning tasks. While previous evaluations have explored whether LLMs can infer intentions or detect deception, they often overlook the individualized reasoning styles that influence how people interpret and act in social contexts. Social deduction games (SDGs) provide a natural testbed for evaluating individualized reasoning styles, where different players may adopt diverse but contextually valid reasoning strategies under identical conditions. To address this, we introduce InMind, a cognitively grounded evaluation framework designed to assess whether LLMs can capture and apply personalized reasoning styles in SDGs. InMind enhances structured gameplay data with round-level strategy traces and post-game reflections, collected under both Observer and Participant modes. It supports four cognitively motivated tasks that jointly evaluate both static alignment and dynamic adaptation. As a case study, we apply InMind to the game Avalon, evaluating 11 state-of-the-art LLMs. General-purpose LLMs, even GPT-4o frequently rely on lexical cues, struggling to anchor reflections in temporal gameplay or adapt to evolving strategies. In contrast, reasoning-enhanced LLMs like DeepSeek-R1 exhibit early signs of style-sensitive reasoning. These findings reveal key limitations in current LLMs' capacity for individualized, adaptive reasoning, and position InMind as a step toward cognitively aligned human-AI interaction.