Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs

TL;DR

This paper develops a probabilistic framework to analyze privacy leakage in AI agents using differential privacy, focusing on enterprise data protection and optimal response generation parameters.

Contribution

It introduces token-level and message-level differential privacy models and derives privacy bounds relating leakage to generation parameters.

Findings

Derived privacy bounds linking leakage to temperature and message length.

Formulated an optimal temperature selection problem for privacy-utility tradeoff.

Provided a framework for analyzing privacy risks in enterprise AI agents.

Abstract

Large language models (LLMs) and AI agents are increasingly integrated into enterprise systems to access internal databases and generate context-aware responses. While such integration improves productivity and decision support, the model outputs may inadvertently reveal sensitive information. Although many prior efforts focus on protecting the privacy of user prompts, relatively few studies consider privacy risks from the enterprise data perspective. Hence, this paper develops a probabilistic framework for analyzing privacy leakage in AI agents based on differential privacy. We model response generation as a stochastic mechanism that maps prompts and datasets to distributions over token sequences. Within this framework, we introduce token-level and message-level differential privacy and derive privacy bounds that relate privacy leakage to generation parameters such as temperature and message length. We further formulate a privacy-utility design problem that characterizes optimal temperature selection.