Membership Inference Attack against Large Language Model-based Recommendation Systems: A New Distillation-based Paradigm

TL;DR

This paper presents a new knowledge distillation-based membership inference attack method tailored for large language model-based recommendation systems, significantly improving attack accuracy over traditional shadow model approaches.

Contribution

Introduces a novel distillation-based MIA paradigm that effectively exploits fused features from a reference model to enhance privacy attack success on LLM recommendation systems.

Findings

Outperforms shadow model-based MIAs and feature-based baselines

Effective on multiple datasets and LLM architectures

Demonstrates practicality for privacy attacks in LLM-driven recommenders

Abstract

Membership Inference Attack (MIA) aims to determine whether a specific data sample was included in the training dataset of a target model. Traditional MIA approaches rely on shadow models to mimic target model behavior, but their effectiveness diminishes for Large Language Model (LLM)-based recommendation systems due to the scale and complexity of training data. This paper introduces a novel knowledge distillation-based MIA paradigm tailored for LLM-based recommendation systems. Our method constructs a reference model via distillation, applying distinct strategies for member and non-member data to enhance discriminative capabilities. The paradigm extracts fused features (e.g., confidence, entropy, loss, and hidden layer vectors) from the reference model to train an attack model, overcoming limitations of individual features. Extensive experiments on extended datasets (Last.FM, MovieLens, Book-Crossing, Delicious) and diverse LLMs (T5, GPT-2, LLaMA3) demonstrate that our approach significantly outperforms shadow model-based MIAs and individual-feature baselines. The results show its practicality for privacy attacks in LLM-driven recommender systems.