Exploring Membership Inference Vulnerabilities in Clinical Large Language Models

TL;DR

This study investigates the privacy vulnerabilities of clinical large language models, revealing limited but present risks of patient data leakage through membership inference attacks, emphasizing the need for enhanced privacy defenses.

Contribution

It provides an empirical analysis of membership inference vulnerabilities in clinical LLMs, introducing domain-specific attack strategies and highlighting areas for privacy improvements.

Findings

Limited but measurable membership leakage detected

Current models show partial resistance to inference attacks

Subtle privacy risks could undermine clinical AI trustworthiness

Abstract

As large language models (LLMs) become progressively more embedded in clinical decision-support, documentation, and patient-information systems, ensuring their privacy and trustworthiness has emerged as an imperative challenge for the healthcare sector. Fine-tuning LLMs on sensitive electronic health record (EHR) data improves domain alignment but also raises the risk of exposing patient information through model behaviors. In this work-in-progress, we present an exploratory empirical study on membership inference vulnerabilities in clinical LLMs, focusing on whether adversaries can infer if specific patient records were used during model training. Using a state-of-the-art clinical question-answering model, Llemr, we evaluate both canonical loss-based attacks and a domain-motivated paraphrasing-based perturbation strategy that more realistically reflects clinical adversarial conditions. Our preliminary findings reveal limited but measurable membership leakage, suggesting that current clinical LLMs provide partial resistance yet remain susceptible to subtle privacy risks that could undermine trust in clinical AI adoption. These results motivate continued development of context-aware, domain-specific privacy evaluations and defenses such as differential privacy fine-tuning and paraphrase-aware training, to strengthen the security and trustworthiness of healthcare AI systems.