Non-Interfering Weight Fields: Treating Model Parameters as a Continuously Extensible Function

TL;DR

This paper introduces Non-Interfering Weight Fields (NIWF), a novel framework that replaces fixed weights with a learned function to prevent catastrophic forgetting in large language models, enabling continual learning without retraining.

Contribution

NIWF offers a structural solution to catastrophic forgetting by using a continuous capability space and snapshotting, allowing models to preserve and extend capabilities without interference.

Findings

Zero forgetting on committed tasks in experiments

Competitive perplexity on new tasks

Enables software-like versioning of neural network capabilities

Abstract

Large language models store all learned knowledge in a single, fixed weight vector. Teaching a model new capabilities requires modifying those same weights, inevitably degrading previously acquired knowledge. This fundamental limitation, known as catastrophic forgetting, has resisted principled solutions for decades. Existing approaches treat weights as immutable artifacts that must be protected through techniques like regularization heuristics, replay buffers, or isolated adapter modules. The problem is none of these provide a structural guarantee against forgetting. In this work, we propose Non-Interfering Weight Fields (NIWF), a framework that replaces the fixed weight paradigm with a learned function that generates weight configurations on demand from a continuous capability coordinate space. After training on a task, we commit the occupied coordinate region by snapshotting the fields outputs on anchor points to enforce a functional lock during all future training. We validate NIWF on sequential instructionfollowing and code generation tasks using Mistral-7B, demonstrating zero forgetting on committed tasks with competitive perplexity on new tasks. The framework introduces the notion of software-like versioning for neural network intelligence, where capabilities can be committed, extended, composed, and rolled back without retraining.