Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm

TL;DR

This paper proposes an experimental framework to evaluate language-based models of human cognitive styles by comparing linguistic features with decision-making behavior, demonstrating moderate-to-high predictive accuracy.

Contribution

It introduces an experimental paradigm for assessing cognitive style models based on language, bridging behavioral science methods with NLP.

Findings

Language features predict decision styles with AUC ~ 0.8

Cognitive styles can be partly inferred from discourse patterns

Experimental validation supports language as a window into cognition

Abstract

While NLP models often seek to capture cognitive states via language, the validity of predicted states is determined by comparing them to annotations created without access the cognitive states of the authors. In behavioral sciences, cognitive states are instead measured via experiments. Here, we introduce an experiment-based framework for evaluating language-based cognitive style models against human behavior. We explore the phenomenon of decision making, and its relationship to the linguistic style of an individual talking about a recent decision they made. The participants then follow a classical decision-making experiment that captures their cognitive style, determined by how preferences change during a decision exercise. We find that language features, intended to capture cognitive style, can predict participants' decision style with moderate-to-high accuracy (AUC ~ 0.8), demonstrating that cognitive style can be partly captured and revealed by discourse patterns.