Understanding Empirical Unlearning with Combinatorial Interpretability

TL;DR

This paper investigates the effectiveness of unlearning methods in neural networks by using combinatorial interpretability to directly inspect knowledge retention and recovery, revealing persistent knowledge despite unlearning efforts.

Contribution

It applies combinatorial interpretability to analyze unlearning in neural networks, providing insights into knowledge persistence and recovery mechanisms.

Findings

Unlearning may inhibit rather than remove knowledge.

Knowledge can often be recovered through fine-tuning.

Interpretability reveals persistent knowledge despite unlearning.

Abstract

While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large foundation models are far from interpretable, understanding whether and how such knowledge persists remains a significant challenge. To address this, we turn to the recently developed framework of combinatorial interpretability. This framework, designed for two-layer neural networks, enables direct inspection of the knowledge encoded in the model weights. We reproduce baseline unlearning methods within the combinatorial interpretability setting and examine their behavior along two dimensions: (i) whether they truly remove knowledge of a target concept (the concept we wish to remove) or merely inhibit its expression while retaining the underlying information, and (ii) how easily the supposedly erased knowledge can be recovered through various fine-tuning operations. Our results shed light within a fully interpretable setting on how knowledge can persist despite unlearning and when it might resurface.