Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models

TL;DR

This paper advocates for shifting from data-point unlearning to knowledge-based unlearning in foundation models, inspired by cognitive science and practical considerations, exemplified through a vision-language model case study.

Contribution

It introduces the concept of knowledge-tracing unlearning for foundation models, addressing practical unlearning challenges and aligning with human cognitive processes.

Findings

Practical unlearning requests are better addressed through knowledge tracing.

Knowledge-tracing unlearning aligns with human forgetting mechanisms.

A case study demonstrates the feasibility of the proposed approach.

Abstract

Machine unlearning removes certain training data points and their influence on AI models (e.g., when a data owner revokes their decision to allow models to learn from the data). In this position paper, we propose to lift data-tracing machine unlearning to knowledge-tracing for foundation models (FMs). We support this position based on practical needs and insights from cognitive studies. Practically, tracing data cannot meet the diverse unlearning requests for FMs, which may be from regulators, enterprise users, product teams, etc., having no access to FMs' massive training data. Instead, it is convenient for these parties to issue an unlearning request about the knowledge or capability FMs (should not) possess. Cognitively, knowledge-tracing unlearning aligns with how the human brain forgets more closely than tracing individual training data points. Finally, we provide a concrete case study about a vision-language FM to illustrate how an unlearner might instantiate the knowledge-tracing machine unlearning paradigm.