Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning

TL;DR

This paper investigates how large language models struggle with overriding pre-trained beliefs when faced with changing rules, and proposes a code-grounded approach that improves their ability to inhibit prior knowledge and reason dynamically.

Contribution

It introduces Code-Grounded Vistas, a fine-tuning method that uses executable code representations to enhance models' capacity to override priors and reason with dynamic rules.

Findings

Larger models can perform worse when suppressing pre-trained associations.

Representing rules as executable code improves prior inhibition.

Code-grounded training outperforms inference-time search methods.

Abstract

LLMs struggle with Semantic Inertia: the inability to inhibit pre-trained priors (e.g., "Lava is Dangerous") when dynamic, in-context rules contradict them. We probe this phenomenon using Baba Is You, where physical laws are mutable text rules, enabling precise evaluation of models' ability to override learned priors when rules change. We quantatively observe that larger models can exhibit inverse scaling: they perform worse than smaller models when natural language reasoning requires suppressing pre-trained associations (e.g., accepting "Lava is Safe"). Our analysis attributes this to natural language encoding, which entangles descriptive semantics and logical rules, leading to persistent hallucinations of familiar physics despite explicit contradictory rules. Here we show that representing dynamics as executable code, rather than descriptive text, reverses this trend and enables effective prior inhibition. We introduce Code-Grounded Vistas (LCV), which fine-tunes models on counterfactual pairs and identifies states with contradictory rules, thereby forcing attention to logical constraints rather than visual semantics. This training-time approach outperforms expensive inference-time search methods in both efficiency and accuracy. Our results demonstrate that representation fundamentally determines whether scaling improves or impairs contextual reasoning. This challenges the assumption that larger models are universally better, with implications for domains that require dynamic overriding of learned priors.