Exploring Concreteness Through a Figurative Lens

TL;DR

This paper investigates how large language models internally represent concreteness, especially in figurative language, revealing a consistent geometric structure that aids in classification and generation steering.

Contribution

It provides a layer-wise and geometric analysis of LLM representations, uncovering a universal concreteness direction useful for figurative language tasks.

Findings

LLMs distinguish literal and figurative noun usage in early layers.

Concreteness is organized along a one-dimensional direction in model representations.

A single concreteness direction enables efficient classification and generation steering.

Abstract

Static concreteness ratings are widely used in NLP, yet a word's concreteness can shift with context, especially in figurative language such as metaphor, where common concrete nouns can take abstract interpretations. While such shifts are evident from context, it remains unclear how LLMs understand concreteness internally. We conduct a layer-wise and geometric analysis of LLM hidden representations across four model families, examining how models distinguish literal vs figurative uses of the same noun and how concreteness is organized in representation space. We find that LLMs separate literal and figurative usage in early layers, and that mid-to-late layers compress concreteness into a one-dimensional direction that is consistent across models. Finally, we show that this geometric structure is practically useful: a single concreteness direction supports efficient figurative-language classification and enables training-free steering of generation toward more literal or more figurative rewrites.