SoK: Semantic Privacy in Large Language Models

TL;DR

This paper systematically analyzes how semantic privacy risks emerge in large language models across their lifecycle, categorizing attack vectors and evaluating current defenses, highlighting critical gaps and future challenges.

Contribution

It introduces a lifecycle-centric framework for analyzing semantic privacy risks in LLMs and assesses the effectiveness of existing defenses, identifying key gaps and open challenges.

Findings

Current defenses inadequately protect against semantic inference attacks.

Significant gaps exist in protecting latent representations and contextual inferences.

Open challenges include quantifying semantic leakage and balancing privacy with generation quality.

Abstract

As Large Language Models (LLMs) are increasingly deployed in sensitive domains, traditional data privacy measures prove inadequate for protecting information that is implicit, contextual, or inferable - what we define as semantic privacy. This Systematization of Knowledge (SoK) introduces a lifecycle-centric framework to analyze how semantic privacy risks emerge across input processing, pretraining, fine-tuning, and alignment stages of LLMs. We categorize key attack vectors and assess how current defenses, such as differential privacy, embedding encryption, edge computing, and unlearning, address these threats. Our analysis reveals critical gaps in semantic-level protection, especially against contextual inference and latent representation leakage. We conclude by outlining open challenges, including quantifying semantic leakage, protecting multimodal inputs, balancing de-identification with generation quality, and ensuring transparency in privacy enforcement. This work aims to inform future research on designing robust, semantically aware privacy-preserving techniques for LLMs.