Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models

TL;DR

This paper introduces an exact, communication-efficient federated unlearning method for ridge regression heads on frozen foundation models, ensuring precise removal of user data influence with minimal retraining cost.

Contribution

It presents a novel protocol for exact federated unlearning in ridge-head models, leveraging sufficient statistics for efficient, exact updates without full retraining.

Findings

Matches centralized ridge retraining accuracy within 10^{-9}

Significantly reduces unlearning cost compared to federated retraining

Supports arbitrary add/delete requests with fixed-size messages

Abstract

Foundation models are commonly deployed as frozen feature extractors with a small trainable head to adapt to private, user-generated data in federated settings. The ``right to be forgotten'' requires removing the influence of specific samples or users from the trained model on demand. Existing federated unlearning methods target general deep models and rely on approximate reconstruction or selective retraining, making exactness costly or elusive. We study this problem in a practically relevant but under-explored regime: a frozen foundation model with a ridge-regression head. The exact optimum depends on the data only through two additive sufficient statistics, which we turn into a communication protocol supporting an arbitrary stream of add and delete requests via fixed-size messages. The server maintains a head that is, in exact arithmetic, pointwise identical to centralized retraining after every request. We provide deterministic retrain-equivalence guarantees, order and partition invariance, two server-side variants, and a Bayesian certificate of zero KL divergence. Experiments on four benchmarks confirm the guarantees: both variants match centralized ridge retraining to within $10^{-9}$ relative Frobenius error and complete each request at orders-of-magnitude lower cost than federated retraining baselines.