Causal Path Alignment: Anchoring the Optimization Trajectory for Controllable In-Parameter Knowledge Editing

TL;DR

This paper introduces Causal Path Alignment (CPA), a novel framework that improves knowledge editing in large language models by anchoring updates to causal pathways, reducing interference and enhancing relation specificity.

Contribution

CPA is a new, relation-aware method that prevents shortcut learning in knowledge editing, ensuring more reliable and context-preserving updates in LLMs.

Findings

CPA eliminates shortcut learning in knowledge editing.

CPA improves relation specificity with minimal side-effects.

CPA is effective across diverse LLM architectures.

Abstract

Knowledge editing is pivotal for efficiently updating the parametric memory of Large Language Models (LLMs), enabling them to function as evolving agents in dynamic environments. However, mainstream in-parameter knowledge editing approaches suffer from Subject-Dominant Memory Interference: modifying a specific fact inadvertently corrupts the broader structural knowledge associated with the same subject within LLMs. We diagnose the root cause as a shortcut learning pathology, where the optimization objective overfits subject representations while bypassing the essential relational context. To rectify this, we propose Causal Path Alignment (CPA), a principled framework designed to anchor the optimization trajectory to valid causal pathways. CPA enforces parameter updates to route through relation-aware intermediate states, thereby preventing the erasure of contextual dependencies. Experimental results across diverse LLM backbones demonstrate that CPA consistently eliminates the shortcut, significantly improving relation specificity while exhibiting minimal side-effects. Moreover, CPA serves as a model-agnostic plug-in for existing editors, paving the way for reliable and trustworthy in-parameter knowledge editing.