CATA: Continual Machine Unlearning via Conflict-Averse Task Arithmetic

TL;DR

This paper introduces CATA, a novel method for continual machine unlearning in vision-language models, effectively removing specific knowledge while preserving overall utility over sequential updates.

Contribution

CATA is the first continual unlearning approach for VLMs that uses conflict-averse task arithmetic to improve forgetting effectiveness and model fidelity.

Findings

CATA outperforms baselines in forgetting effectiveness.

CATA maintains higher model utility after unlearning.

CATA prevents re-emergence of forgotten knowledge over time.

Abstract

Vision-language models (VLMs) have shown remarkable ability in aligning visual and textual representations, enabling a wide range of multimodal applications. However, their large-scale training data inevitably raises concerns about privacy, copyright, and undesirable content, creating a strong need for machine unlearning. While existing studies mainly focus on single-shot unlearning, practical VLM deployment often involves sequential removal requests over time, giving rise to continual machine unlearning. In this work, we make the first attempt to study continual unlearning for VLMs and identify three key challenges in this setting: effectiveness in removing target knowledge, fidelity in preserving retained model utility, and persistence in preventing knowledge re-emergence under sequential updates. To address these challenges, we propose CATA, a conflict-averse task arithmetic method that represents each forget request as an unlearning task vector. By maintaining historical task vectors and performing sign-aware conflict-averse aggregation, CATA suppresses conflicting update components that may weaken previous forgetting effects. Extensive experiments under both single-shot and continual settings show that CATA outperforms baselines in terms of forgetting effectiveness, model fidelity, and forgetting persistence.