Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuning

TL;DR

This paper reveals that in lifelong fine-tuning of large language models, coverage reliability can deteriorate faster than accuracy, and introduces a calibration replay method to maintain coverage effectively.

Contribution

It demonstrates coverage collapse occurs before accuracy drops and proposes a lightweight calibration replay technique to preserve coverage during continual learning.

Findings

Coverage loss exceeds accuracy loss by about 3.4 times on average.

Coverage can drop from 0.92 to 0.61 while accuracy remains stable.

Calibration replay restores coverage within two points of the nominal level.

Abstract

Continual learning for large language models is typically evaluated through accuracy retention under sequential fine-tuning. We argue that this perspective is incomplete, because uncertainty reliability can degrade earlier and more sharply than top-1 performance. We study this empirically by measuring conformal coverage and calibration error on sequentially fine-tuned models across three model families and eight task sequences drawn primarily from classification and multiple-choice benchmarks. Across the classification-style settings we study, coverage loss exceeds accuracy loss by a factor of roughly \(3.4\times \pm 0.5\times\) on average across seeds; in the most pronounced case, coverage drops from \(0.92\) to \(0.61\), while accuracy remains within three points of baseline. Standard continual-learning methods that preserve accuracy do not automatically preserve coverage, and naive calibration baselines recover only part of the gap. We propose calibration replay, a lightweight post-hoc procedure that maintains a task-specific held-out buffer and refits a task-specific conformal threshold under the current model after each update. It adds no training-time gradient cost, uses less than one percent of the memory of ordinary experience replay, and typically restores coverage to within two points of nominal at buffer size \(m = 200\). We accompany the empirical study with a drift decomposition, a finite-sample recovery theorem showing exact conformal validity under exchangeability, and a mixture-validity proposition explaining why pooled thresholds do not suffice. Our guarantees are stated for classification-style tasks with task-specific buffers; extensions to open-ended generation are exploratory.