De-identification is not enough: a comparison between de-identified and synthetic clinical notes

TL;DR

This paper compares de-identified and synthetically generated clinical notes, revealing that synthetic notes can match real data in utility but pose similar privacy risks, highlighting the need for better privacy-preserving methods.

Contribution

It introduces a novel method for generating synthetic clinical notes using large language models and evaluates their privacy and utility in clinical tasks.

Findings

De-identification does not prevent membership inference attacks.

Synthetic notes can perform similarly to real data in clinical tasks.

Synthetic notes pose comparable privacy risks to real data.

Abstract

For sharing privacy-sensitive data, de-identification is commonly regarded as adequate for safeguarding privacy. Synthetic data is also being considered as a privacy-preserving alternative. Recent successes with numerical and tabular data generative models and the breakthroughs in large generative language models raise the question of whether synthetically generated clinical notes could be a viable alternative to real notes for research purposes. In this work, we demonstrated that (i) de-identification of real clinical notes does not protect records against a membership inference attack, (ii) proposed a novel approach to generate synthetic clinical notes using the current state-of-the-art large language models, (iii) evaluated the performance of the synthetically generated notes in a clinical domain task, and (iv) proposed a way to mount a membership inference attack where the target model is trained with synthetic data. We observed that when synthetically generated notes closely match the performance of real data, they also exhibit similar privacy concerns to the real data. Whether other approaches to synthetically generated clinical notes could offer better trade-offs and become a better alternative to sensitive real notes warrants further investigation.