Geometric Organization of Cognitive States in Transformer Embedding Spaces

TL;DR

This paper demonstrates that transformer sentence embeddings encode structured geometric information aligned with human cognitive attributes, as shown through quantitative decoding and qualitative visualization analyses.

Contribution

It provides evidence that transformer embeddings contain interpretable cognitive structure, validated by statistical tests and visualization techniques.

Findings

Transformer embeddings reliably decode cognitive annotation scores.

Linear probes capture substantial geometric structure in embeddings.

Embedding spaces show a low-to-high cognitive gradient with local confusions.

Abstract

Recent work has shown that transformer-based language models learn rich geometric structure in their embedding spaces. In this work, we investigate whether sentence embeddings exhibit structured geometric organization aligned with human-interpretable cognitive or psychological attributes. We construct a dataset of 480 natural-language sentences annotated with both continuous energy scores (ranging from -5 to +5) and discrete tier labels spanning seven ordered cognitive annotation tiers, intended to capture a graded progression from highly constricted or reactive expressions toward more coherent and integrative cognitive states. Using fixed sentence embeddings from multiple transformer models, we evaluate the recoverability of these annotations via linear and shallow nonlinear probes. Across models, both continuous energy scores and tier labels are reliably decodable, with linear probes already capturing substantial structure. To assess statistical significance, we conduct nonparametric permutation tests that randomize labels, showing that probe performance exceeds chance under both regression and classification null hypotheses. Qualitative analyses using UMAP visualizations and tier-level confusion matrices further reveal a coherent low-to-high gradient and predominantly local (adjacent-tier) confusions. Together, these results indicate that transformer embedding spaces exhibit statistically significant geometric organization aligned with the annotated cognitive structure.