Learning with Recoverable Forgetting

TL;DR

This paper introduces LIRF, a novel learning scheme that enables explicit, efficient, and reversible removal and recovery of specific knowledge in neural networks, addressing privacy and control concerns in lifelong learning.

Contribution

LIRF is the first method to explicitly handle task- or sample-specific knowledge removal and recovery through knowledge deposit and withdrawal schemes.

Findings

LIRF effectively isolates and recovers knowledge with minimal fine-tuning.

The method demonstrates strong generalization across multiple datasets.

LIRF is efficient in terms of data and computational resources.

Abstract

Life-long learning aims at learning a sequence of tasks without forgetting the previously acquired knowledge. However, the involved training data may not be life-long legitimate due to privacy or copyright reasons. In practical scenarios, for instance, the model owner may wish to enable or disable the knowledge of specific tasks or specific samples from time to time. Such flexible control over knowledge transfer, unfortunately, has been largely overlooked in previous incremental or decremental learning methods, even at a problem-setup level. In this paper, we explore a novel learning scheme, termed as Learning wIth Recoverable Forgetting (LIRF), that explicitly handles the task- or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal, which allow for isolating user-designated knowledge from a pre-trained network and injecting it back when necessary. During the knowledge deposit process, the specified knowledge is extracted from the target network and stored in a deposit module, while the insensitive or general knowledge of the target network is preserved and further augmented. During knowledge withdrawal, the taken-off knowledge is added back to the target network. The deposit and withdraw processes only demand for a few epochs of finetuning on the removal data, ensuring both data and time efficiency. We conduct experiments on several datasets, and demonstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.