Layerwise Recall and the Geometry of Interwoven Knowledge in LLMs

TL;DR

This paper investigates how large language models encode scientific knowledge, revealing a geometric 3D spiral structure in hidden states that reflects the organization of the periodic table, and analyzing layer-specific encoding of attributes.

Contribution

It uncovers a geometric spiral structure in LLM hidden states and details how different layers encode attributes and categorical distinctions, advancing understanding of knowledge representation.

Findings

Hidden states form a 3D spiral structure aligned with the periodic table

Middle layers encode continuous, overlapping attributes for indirect recall

Deeper layers sharpen categorical distinctions and incorporate linguistic context

Abstract

This study explores how large language models (LLMs) encode interwoven scientific knowledge, using chemical elements and LLaMA-series models as a case study. We identify a 3D spiral structure in the hidden states that aligns with the conceptual structure of the periodic table, suggesting that LLMs can reflect the geometric organization of scientific concepts learned from text. Linear probing reveals that middle layers encode continuous, overlapping attributes that enable indirect recall, while deeper layers sharpen categorical distinctions and incorporate linguistic context. These findings suggest that LLMs represent symbolic knowledge not as isolated facts, but as structured geometric manifolds that intertwine semantic information across layers. We hope this work inspires further exploration of how LLMs represent and reason about scientific knowledge, particularly in domains such as materials science.