A Comprehensive Evaluation of LLM Unlearning Robustness under Multi-Turn Interaction

TL;DR

This paper evaluates the robustness of language model unlearning in multi-turn interactions, revealing that models often recover forgotten knowledge during interaction, which challenges static evaluation methods.

Contribution

It introduces the first comprehensive study of LLM unlearning robustness in interactive settings, highlighting the limitations of static evaluation and proposing the need for dynamic assessment methods.

Findings

Knowledge can be recovered through interaction despite unlearning.

Stronger unlearning can cause behavioral rigidity, not genuine forgetting.

Static evaluations may overestimate unlearning effectiveness.

Abstract

Machine unlearning aims to remove the influence of specific training data from pre-trained models without retraining from scratch, and is increasingly important for large language models (LLMs) due to safety, privacy, and legal concerns. Although prior work primarily evaluates unlearning in static, single-turn settings, forgetting robustness under realistic interactive use remains underexplored. In this paper, we study whether unlearning remains stable in interactive environments by examining two common interaction patterns: self-correction and dialogue-conditioned querying. We find that knowledge appearing forgotten in static evaluation can often be recovered through interaction. Although stronger unlearning improves apparent robustness, it often results in behavioral rigidity rather than genuine knowledge erasure. Our findings suggest that static evaluation may overestimate real-world effectiveness and highlight the need for ensuring stable forgetting under interactive settings.