When Models Ignore Definitions: Measuring Semantic Override Hallucinations in LLM Reasoning

TL;DR

This paper investigates how large language models often fail to correctly interpret locally redefined semantics in logic and circuit reasoning tasks, revealing a gap between their surface correctness and true understanding.

Contribution

The study introduces a benchmark for testing semantic override in LLMs and systematically analyzes their failures in local semantic redefinition scenarios.

Findings

LLMs frequently ignore local redefinitions in logic tasks

Models often make confident but incompatible assumptions

LLMs drop constraints even in simple formal reasoning settings

Abstract

Large language models (LLMs) demonstrate strong performance on standard digital logic and Boolean reasoning tasks, yet their reliability under locally redefined semantics remains poorly understood. In many formal settings, such as circuit specifications, examinations, and hardware documentation, operators and components are explicitly redefined within narrow scope. Correct reasoning in these contexts requires models to temporarily suppress globally learned conventions in favor of prompt-local definitions. In this work, we study a systematic failure mode we term semantic override, in which an LLM reverts to its pretrained default interpretation of operators or gate behavior despite explicit redefinition in the prompt. We also identify a related class of errors, assumption injection, where models commit to unstated hardware semantics when critical details are underspecified, rather than requesting clarification. We introduce a compact micro-benchmark of 30 logic and digital-circuit reasoning tasks designed as verifier-style traps, spanning Boolean algebra, operator overloading, redefined gates, and circuit-level semantics. Evaluating three frontier LLMs, we observe persistent noncompliance with local specifications, confident but incompatible assumptions, and dropped constraints even in elementary settings. Our findings highlight a gap between surface-level correctness and specification-faithful reasoning, motivating evaluation protocols that explicitly test local unlearning and semantic compliance in formal domains.