Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models

TL;DR

This paper investigates methods to detect memorized data in large language models used for recommendations, comparing manual and automated prompt techniques, and finds automated prompt engineering most promising.

Contribution

It introduces and evaluates three approaches—jailbreak prompts, unsupervised probing, and automatic prompt engineering—for detecting memorized data in LLMs, highlighting the potential of automated methods.

Findings

Jailbreak prompts do not reliably improve memorized data retrieval.

Unsupervised probing distinguishes genuine from fabricated data but struggles with numerical info.

Automated prompt engineering shows moderate success in extracting memorized items.

Abstract

Large Language Models (LLMs) are increasingly applied in recommendation scenarios due to their strong natural language understanding and generation capabilities. However, they are trained on vast corpora whose contents are not publicly disclosed, raising concerns about data leakage. Recent work has shown that the MovieLens-1M dataset is memorized by both the LLaMA and OpenAI model families, but the extraction of such memorized data has so far relied exclusively on manual prompt engineering. In this paper, we pose three main questions: Is it possible to enhance manual prompting? Can LLM memorization be detected through methods beyond manual prompting? And can the detection of data leakage be automated? To address these questions, we evaluate three approaches: (i) jailbreak prompt engineering; (ii) unsupervised latent knowledge discovery, probing internal activations via Contrast-Consistent Search (CCS) and Cluster-Norm; and (iii) Automatic Prompt Engineering (APE), which frames prompt discovery as a meta-learning process that iteratively refines candidate instructions. Experiments on MovieLens-1M using LLaMA models show that jailbreak prompting does not improve the retrieval of memorized items and remains inconsistent; CCS reliably distinguishes genuine from fabricated movie titles but fails on numerical user and rating data; and APE retrieves item-level information with moderate success yet struggles to recover numerical interactions. These findings suggest that automatically optimizing prompts is the most promising strategy for extracting memorized samples.