Holistic Audit Dataset Generation for LLM Unlearning via Knowledge Graph Traversal and Redundancy Removal

TL;DR

This paper introduces HANKER, an automated framework that generates comprehensive audit datasets for LLM unlearning by leveraging knowledge graphs, significantly improving detection of knowledge memorization and addressing redundancy issues.

Contribution

HANKER systematically creates large, fine-grained audit datasets for LLM unlearning evaluation, overcoming limitations of existing benchmarks and revealing the impact of knowledge redundancy.

Findings

Generated over 69,000 and 111,000 audit cases for News and Books datasets.

Redundancy inflates unlearning effectiveness metrics like ROUGE and Entailment Scores.

Systematic deduplication is essential for accurate unlearning assessment.

Abstract

In recent years, Large Language Models (LLMs) have faced increasing demands to selectively remove sensitive information, protect privacy, and comply with copyright regulations through unlearning, by Machine Unlearning. While evaluating unlearning effectiveness is crucial, existing benchmarks are limited in scale and comprehensiveness, typically containing only a few hundred test cases. We identify two critical challenges in generating holistic audit datasets: ensuring audit adequacy and handling knowledge redundancy between forget and retain dataset. To address these challenges, we propose HANKER, an automated framework for holistic audit dataset generation leveraging knowledge graphs to achieve fine-grained coverage and eliminate redundant knowledge. Applying HANKER to the popular MUSE benchmark, we successfully generated over 69,000 and 111,000 audit cases for the News and Books datasets respectively, identifying thousands of knowledge memorization instances that the previous benchmark failed to detect. Our empirical analysis uncovers how knowledge redundancy significantly skews unlearning effectiveness metrics, with redundant instances artificially inflating the observed memorization measurements ROUGE from 19.7% to 26.1% and Entailment Scores from 32.4% to 35.2%, highlighting the necessity of systematic deduplication for accurate assessment.