Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models

TL;DR

This paper introduces Min-K%++, a theoretically grounded method for detecting pre-training data in large language models, significantly outperforming previous heuristics and state-of-the-art techniques across multiple benchmarks.

Contribution

The paper proposes a novel, theoretically motivated approach for pre-training data detection based on local maxima identification in LLMs, advancing beyond heuristic-based methods.

Findings

Achieves new state-of-the-art detection performance on WikiMIA benchmark.

Consistently outperforms reference-free methods on MIMIR benchmark.

Performs comparably to reference-based methods requiring extra models.

Abstract

The problem of pre-training data detection for large language models (LLMs) has received growing attention due to its implications in critical issues like copyright violation and test data contamination. Despite improved performance, existing methods (including the state-of-the-art, Min-K%) are mostly developed upon simple heuristics and lack solid, reasonable foundations. In this work, we propose a novel and theoretically motivated methodology for pre-training data detection, named Min-K%++. Specifically, we present a key insight that training samples tend to be local maxima of the modeled distribution along each input dimension through maximum likelihood training, which in turn allow us to insightfully translate the problem into identification of local maxima. Then, we design our method accordingly that works under the discrete distribution modeled by LLMs, whose core idea is to determine whether the input forms a mode or has relatively high probability under the conditional categorical distribution. Empirically, the proposed method achieves new SOTA performance across multiple settings. On the WikiMIA benchmark, Min-K%++ outperforms the runner-up by 6.2% to 10.5% in detection AUROC averaged over five models. On the more challenging MIMIR benchmark, it consistently improves upon reference-free methods while performing on par with reference-based method that requires an extra reference model.