Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models

TL;DR

This study systematically investigates how and where in transformer models various psycholinguistic features are encoded, revealing that encoding location depends on the probing method and model architecture, with shared depth patterns across models.

Contribution

It provides a comprehensive layer-wise analysis of psycholinguistic feature encoding across multiple transformer architectures and probing methods, highlighting method-dependent localization and shared depth patterns.

Findings

Contextualized embeddings improve feature selectivity.

Final-layer representations are rarely optimal for psycholinguistic info.

Lexical features peak earlier; experiential and affective features peak later.

Abstract

Understanding where transformer language models encode psychologically meaningful aspects of meaning is essential for both theory and practice. We conduct a systematic layer-wise probing study of 58 psycholinguistic features across 10 transformer models, spanning encoder-only and decoder-only architectures, and compare three embedding extraction methods. We find that apparent localization of meaning is strongly method-dependent: contextualized embeddings yield higher feature-specific selectivity and different layer-wise profiles than isolated embeddings. Across models and methods, final-layer representations are rarely optimal for recovering psycholinguistic information with linear probes. Despite these differences, models exhibit a shared depth ordering of meaning dimensions, with lexical properties peaking earlier and experiential and affective dimensions peaking later. Together, these results show that where meaning "lives" in transformer models reflects an interaction between methodological choices and architectural constraints.