SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations

TL;DR

This paper systematically analyzes privacy threats and defenses for large language models in healthcare, highlighting challenges across data processing, fine-tuning, and inference phases, and proposing future research directions.

Contribution

It provides a comprehensive threat model and evaluates existing privacy-preserving techniques for LLMs in healthcare, identifying limitations and offering phase-aware recommendations.

Findings

Existing defenses have limitations in diverse healthcare settings.

Threat landscape varies across LLM phases: data, fine-tuning, inference.

Recommendations aim to enhance privacy guarantees in regulated environments.

Abstract

Large Language Models (LLMs) are increasingly adopted in healthcare to support clinical decision-making, summarize electronic health records (EHRs), and enhance patient care. However, this integration introduces significant privacy and security challenges, driven by the sensitivity of clinical data and the high-stakes nature of medical workflows. These risks become even more pronounced across heterogeneous deployment environments, ranging from small on-premise hospital systems to regional health networks, each with unique resource limitations and regulatory demands. This Systematization of Knowledge (SoK) examines the evolving threat landscape across the three core LLM phases: Data preprocessing, Fine-tuning, and Inference within realistic healthcare settings. We present a detailed threat model that characterizes adversaries, capabilities, and attack surfaces at each phase, and we systematize how existing privacy-preserving techniques (PPTs) attempt to mitigate these vulnerabilities. While existing defenses show promise, our analysis identifies persistent limitations in securing sensitive clinical data across diverse operational tiers. We conclude with phase-aware recommendations and future research directions aimed at strengthening privacy guarantees for LLMs in regulated environments. This work provides a foundation for understanding the intersection of LLMs, threats, and privacy in healthcare, offering a roadmap toward more robust and clinically trustworthy AI systems.