EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure

TL;DR

EASE is a novel federated multimodal unlearning framework that effectively isolates and removes forgotten knowledge by addressing cross-modal entanglement and residual anchors, improving unlearning accuracy.

Contribution

The paper introduces EASE, a unified method that closes residual anchors in federated multimodal models, enabling more effective and precise unlearning of forgotten information.

Findings

EASE achieves near-retraining performance, with only 0.2 and 4.2 R@1 points difference on Flickr30K.

It effectively isolates forget-exclusive directions from shared support.

EASE outperforms previous methods across multiple datasets and unlearning scenarios.

Abstract

Federated Multimodal Learning (FML) trains multimodal models across decentralized clients while keeping their image-text pairs private. However, joint embedding training entangles forgotten knowledge across both modalities and client gradient subspaces, hindering federated unlearning. Previous federated unlearning approaches neither sever the cross-modal reconstruction channel mediated by bilinear coupling nor separate forget-exclusive update directions from those shared with retained clients. We identify an Anchor Principle for federated multimodal contrastive unlearning: forgotten alignments persist through three residual anchors arising from bilinear cross-modal coupling, principal-angle subspace entanglement, and continued federated updates. At the modality level, we show that bilateral displacement of both visual and language branches closes the cross-modal reconstruction channel. Correspondingly, our method addresses subspace entanglement through Cosine--Sine decomposition of client-update subspaces, isolating forget-exclusive directions from retain support. Moreover, we propose a direction-selective Forget Lock that bounds residual drift across rounds. Combining these strategies, we present EASE, an Entanglement-Aware Subspace Excision framework that closes all three anchor channels under a unified design. EASE demonstrates consistent superiority across multiple datasets and unlearning scenarios, for instance, matching the retrain reference to within 0.2 and 4.2 R@1 points on the forget and retain sides under client unlearning on Flickr30K with CLIP-B/32.