Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning

TL;DR

This paper introduces RASLIK, a scalable randomized retrieval algorithm that improves data unlearning in large language models by balancing forgetting and retention more effectively.

Contribution

It proposes RASLIK, a novel randomized antipodal search method that enhances retrieval for unlearning, outperforming deterministic baselines and oracle sampling.

Findings

RASLIK reduces selection variance and achieves sublinear complexity.

RASLIK consistently outperforms deterministic baselines and oracle sampling.

The method improves the trade-off between forgetting and retention in LLM unlearning.

Abstract

Large language models (LLMs) sometimes memorize undesirable knowledge, which must be removed after deployment. Prior work on machine unlearning has focused largely on optimization methods that adjust parameters to enforce forgetting while preserving retention. However, these approaches assume that the forget and retain sets are readily available, which rarely holds in practice. Unlearning is typically triggered by an undesired generation at inference time, making the retrieval of relevant data the central challenge. We introduce the notion of data Pareto improvement for LLM unlearning, which formalizes how retrieval can expand the achievable trade-off frontier between forgetting and retention. To realize this principle, we propose Randomized Antipodal Search on Linearized Influence Kernel (RASLIK), a retrieval algorithm that combines permutation-projection hashing with randomized antipodal search. RASLIK reduces selection variance, achieves sublinear complexity, and yields a double gain in both quality and efficiency. Across multiple models, datasets, and unlearning algorithms, RASLIK consistently outperforms deterministic baselines and even oracle sampling, establishing randomized search as a principled and scalable solution for data-centric unlearning.