In Machina N400: Pinpointing Where a Causal Language Model Detects Semantic Violations

TL;DR

This paper investigates where and how a causal language model detects semantic violations, revealing that detection accuracy peaks in middle layers and that violations initially expand then condense the representational space.

Contribution

It introduces a layered analysis of semantic violation detection in transformer models, highlighting the dynamic encoding process across layers.

Findings

Detection accuracy peaks in middle layers

Violation encoding involves initial expansion then collapse of representational space

Semantic anomalies are detected after syntactic resolution, similar to human reading

Abstract

How and where does a transformer notice that a sentence has gone semantically off the rails? To explore this question, we evaluated the causal language model (phi-2) using a carefully curated corpus, with sentences that concluded plausibly or implausibly. Our analysis focused on the hidden states sampled at each model layer. To investigate how violations are encoded, we utilized two complementary probes. First, we conducted a per-layer detection using a linear probe. Our findings revealed that a simple linear decoder struggled to distinguish between plausible and implausible endings in the lowest third of the model's layers. However, its accuracy sharply increased in the middle blocks, reaching a peak just before the top layers. Second, we examined the effective dimensionality of the encoded violation. Initially, the violation widens the representational subspace, followed by a collapse after a mid-stack bottleneck. This might indicate an exploratory phase that transitions into rapid consolidation. Taken together, these results contemplate the idea of alignment with classical psycholinguistic findings in human reading, where semantic anomalies are detected only after syntactic resolution, occurring later in the online processing sequence.