Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

TL;DR

This paper investigates the privacy vulnerabilities of large language models aligned with human preferences, revealing that models trained with DPO are more susceptible to membership inference attacks than those trained with PPO.

Contribution

It provides a theoretical analysis of privacy risks in preference-based LLM alignment and introduces PREMIA, a novel attack framework for assessing these vulnerabilities.

Findings

DPO models are more vulnerable to membership inference attacks than PPO models

PREMIA effectively exposes privacy gaps in preference data aligned LLMs

Empirical results confirm heightened vulnerability of DPO models

Abstract

Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA.