Extracting Training Dialogue Data from Large Language Model based Task Bots

TL;DR

This paper investigates privacy risks in LLM-based task-oriented dialogue systems by evaluating data extraction attacks, proposing novel methods, and analyzing factors influencing memorization to improve understanding and mitigation of data leakage.

Contribution

It introduces tailored attack techniques for LLM-based dialogue systems, evaluates their effectiveness, and analyzes memorization factors to inform privacy mitigation strategies.

Findings

Proposed attack achieves over 70% precision in extracting dialogue training labels.

Analyzed key factors influencing LLM memorization in dialogue systems.

Provided insights into privacy risks and mitigation strategies for LLM-based TODS.

Abstract

Large Language Models (LLMs) have been widely adopted to enhance Task-Oriented Dialogue Systems (TODS) by modeling complex language patterns and delivering contextually appropriate responses. However, this integration introduces significant privacy risks, as LLMs, functioning as soft knowledge bases that compress extensive training data into rich knowledge representations, can inadvertently memorize training dialogue data containing not only identifiable information such as phone numbers but also entire dialogue-level events like complete travel schedules. Despite the critical nature of this privacy concern, how LLM memorization is inherited in developing task bots remains unexplored. In this work, we address this gap through a systematic quantitative study that involves evaluating existing training data extraction attacks, analyzing key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposing novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference. Experimental results demonstrate the effectiveness of our proposed data extraction attack. Our method can extract thousands of training labels of dialogue states with best-case precision exceeding 70%. Furthermore, we provide an in-depth analysis of training data memorization in LLM-based TODS by identifying and quantifying key influencing factors and discussing targeted mitigation strategies.