TL;DR
This paper introduces the Lattice Representation Hypothesis, proposing that large language models encode conceptual hierarchies and logical operations within embedding geometry, bridging continuous and symbolic reasoning.
Contribution
It unifies the Linear Representation Hypothesis with Formal Concept Analysis, demonstrating how embeddings encode concept lattices and enabling geometric symbolic reasoning.
Findings
Embeddings encode concept lattices and logical structures.
Linear attribute directions induce a concept lattice.
Empirical evidence from WordNet supports the hypothesis.
Abstract
We propose the Lattice Representation Hypothesis of large language models: a symbolic backbone that grounds conceptual hierarchies and logical operations in embedding geometry. Our framework unifies the Linear Representation Hypothesis with Formal Concept Analysis (FCA), showing that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections. This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations, and admits a canonical form when attribute directions are linearly independent. Experiments on WordNet sub-hierarchies provide empirical evidence that LLM embeddings encode concept lattices and their logical structure, revealing a principled bridge between continuous geometry and symbolic abstraction.
Peer Reviews
Decision·ICLR 2026 Poster
1. I think the authors are targeting an important and ambitious question to resolve the conflicts between continuous representation space in neural language models and symbolic abstractions. While I am not an expert in the related set theory to judge the rigor and correctness of the framework, I appreciate the principled approach. 2. While limited, the authors attempted to empirically validate their theoretical statements with the controlled dataset. 3. The paper is generally well written,
1. I am unclear about the WordNet experiment; how exactly is the $v_g$? Is it simply a token-embedding output? While the theoretical framework is presented as a general account of “the hidden lattice geometry of LLMs”, I am concerned that the empirical experiments with WordNet appear to rely exclusively on static token embeddings. It is unclear if the experiment concerns contextual forward passes or layer-wise activations. Consequently, the experiments probe the lexical geometry of the embedding
Well organized, thorough treatment of related work and preliminaries, easy to follow, clear Novel combination of the Linear Representation Hypothesis with a formal, well-defined framework The claims introduced in the theoretical section (Section 3) are substantiated by the lattice extraction results from the experiment section (Section 4) Very good reproducibility information, particularly in section 4.1
Limited number of datasets and LLMs used in the evaluation (3 datasets, 3 LLMs) Little to no discussion on which types of concepts (e.g., those arranged on a curved manifold such as months of the year or days of the week) could or could not be represented by this framework. Somewhat recent work has indicated that “Not All Language Model Features are One-Dimensionally Linear” (Engels et al 2025)
The paper provides a valuable theoretical connection between the extensional view of concepts and a more structured intensional view based on Formal Concept Analysis (FCA). This shift from "sets of tokens" to "intersections of attributes" is both compelling and logically sound. The experimental results, particularly those reported in Tables 1 and 2, are strong. The high F1 scores provide substantial empirical evidence that the "half-space" and "projection-profile" models are not merely theoreti
1. Missing Geometric Visualization: The paper’s title and central thesis emphasize "lattice geometry." While Figure 2(b) presents an excellent illustration of this idea, the paper does not provide an equivalent visualization for real, recovered data. We observe 1D projection distributions (Figure 3), but not the actual geometric arrangement of attribute directions and concept embeddings (e.g., via PCA or a similar projection) to visually confirm the half-space intersections. This represents a si
Videos
Taxonomy
TopicsTopic Modeling · Advanced Graph Neural Networks · Sentiment Analysis and Opinion Mining
