Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attack

TL;DR

This paper introduces Tag&Tab, a novel method leveraging keyword tagging and likelihood analysis to improve detection of pretraining data in large language models, significantly outperforming existing approaches.

Contribution

The paper presents a new keyword-based membership inference attack method, Tag&Tab, that enhances accuracy in detecting LLM training data by incorporating semantic importance of words.

Findings

Achieved 5.3% to 17.6% higher AUC scores than prior methods.

Demonstrated effectiveness across four benchmark datasets and various LLM sizes.

Highlights the importance of word significance in membership inference attacks.

Abstract

Large language models (LLMs) have become essential tools for digital task assistance. Their training relies heavily on the collection of vast amounts of data, which may include copyright-protected or sensitive information. Recent studies on detecting pretraining data in LLMs have primarily focused on sentence- or paragraph-level membership inference attacks (MIAs), usually involving probability analysis of the target model's predicted tokens. However, these methods often exhibit poor accuracy, failing to account for the semantic importance of textual content and word significance. To address these shortcomings, we propose Tag&Tab, a novel approach for detecting data used in LLM pretraining. Our method leverages established natural language processing (NLP) techniques to tag keywords in the input text, a process we term Tagging. Then, the LLM is used to obtain probabilities for these keywords and calculate their average log-likelihood to determine input text membership, a process we refer to as Tabbing. Our experiments on four benchmark datasets (BookMIA, MIMIR, PatentMIA, and the Pile) and several open-source LLMs of varying sizes demonstrate an average increase in AUC scores ranging from 5.3% to 17.6% over state-of-the-art methods. Tag&Tab not only sets a new standard for data leakage detection in LLMs, but its outstanding performance is a testament to the importance of words in MIAs on LLMs.