Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablation

TL;DR

This paper introduces MechaRule, a novel method for grounding rule extraction in large language model circuits by localizing sparse neurons called agonists, improving interpretability and robustness.

Contribution

MechaRule provides an efficient, data-driven approach to identify neurons responsible for specific behaviors in LLMs, connecting mechanistic interpretability with symbolic rule extraction.

Findings

MechaRule recalls 96.8% of high-effect agonists in experiments.

Suppressing localized agonists reduces arithmetic accuracy by up to 71.1%.

Suppressing localized agonists reduces jailbreak success by up to 8.8%.

Abstract

A key goal of explainable AI (XAI) is to express the decision logic of large language models (LLMs) in symbolic form and link it to internal mechanisms. Global rule-extraction methods typically learn symbolic surrogates without grounding rules in model circuitry, while mechanistic interpretability can connect behaviors to neuron sets but often depends on hand-crafted hypotheses and expensive neuron-level interventions. We introduce MechaRule, a pipeline that grounds rule extraction in LLM circuits by efficiently localizing sparse neurons called agonists, whose activation neutralization disrupts rule-related behaviors. MechaRule rests on two empirical observations. First, within a fixed baseline/flip regime, sparse agonist effects can be approximately monotone and saturating: a few dominant neuron activations can overtop weaker ones at coarse scales, while overlapping neurons flip many of the same examples. This motivates viewing localization as adaptive group testing driven by a regime-conditional strength predicate with confidence-guided conservative pruning, yielding Theta(k log(N/k) + k) interventions over N candidates when k << N neurons are agonists under the monotone-overtopping abstraction. Second, agonists emerge more reliably when ablations are verified through data splits aligned with close-to-faithful rule behavior; spectral splits remain a useful rule-free fallback, while unfaithful splits degrade localization. Empirically, overtopping appears mainly in learned, task-aligned regimes: on arithmetic and jailbreak tasks across Qwen2 and GPT-J, MechaRule recalls 96.8% of high-effect brute-force agonists in completed comparisons, and suppressing localized agonists reduces arithmetic accuracy and jailbreak success by up to 71.1% and 8.8%, respectively.