Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

TL;DR

This paper introduces a geometric perspective on LLM continual post-training, identifying geometry conflict as a key factor in forgetting and proposing a new method, GCWM, to improve knowledge retention without replay data.

Contribution

It presents a novel geometry-based framework for understanding and controlling forgetting in LLM continual post-training, along with a data-free update method called GCWM.

Findings

GCWM outperforms data-free baselines in continual learning tasks.

Forgetting correlates with geometry misalignment between tasks and model state.

Compatibility of sequential updates predicts transfer success or interference.

Abstract

Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability transfer and when they cause catastrophic forgetting. Existing methods mitigate forgetting through sequential fine-tuning, replay, regularization, or model merging, but offer limited criteria for determining when incorporating new updates is beneficial or harmful. In this work, we study LLM continual post-training through three questions: What drives forgetting? When do sequentially acquired capabilities transfer or interfere? How can compatibility be used to control update integration? We address these questions through task geometry: we represent each post-training task by its parameter update and study the covariance geometry induced by the update. Our central finding is that: forgetting can be considered as a state-relative update-integration failure, it arises when the covariance geometries induced by tasks misalign with the geometry of the evolving model state. Sequential updates transfer when they remain compatible with the model state shaped by previous updates, and interfere when state-relative geometry conflict becomes high. Motivated by this finding, we propose Geometry-Conflict Wasserstein Merging (GCWM), a data-free update-integration method that constructs a shared Wasserstein metric via Gaussian Wasserstein barycenters and uses geometry conflict to gate geometry-aware correction. Across Qwen3 0.6B--14B on domain-continual and capability-continual settings, GCWM consistently outperforms data-free baselines, improving retention and final performance without replay data. These results identify geometry conflict as both an explanatory signal for forgetting and a practical control signal for LLM continual post-training.