Guardians of the data: NER and LLMs for effective medical record anonymization in Brazilian Portuguese

TL;DR

This paper introduces AnonyMed-BR, a new dataset for anonymizing medical records in Brazilian Portuguese, and evaluates extractive and generative models for effective anonymization.

Contribution

AnonyMed-BR is the first manually annotated anonymization dataset for Brazilian Portuguese medical texts.

Findings

Both extractive and generative models achieved F1 scores above 0.90 in anonymizing sensitive entities.

Including synthetic data improved model generalization and task-specific fine-tuning outperformed biomedical pre-training.

The dataset and methodology enable privacy-preserving NLP research in Brazilian healthcare.

Abstract

The anonymization of medical records is essential to protect patient privacy while enabling the use of clinical data for research and Natural Language Processing (NLP) applications. However, for Brazilian Portuguese, the lack of publicly available and high-quality anonymized datasets limits progress in this area.

In this study, we present AnonyMed-BR, a novel dataset of Brazilian medical records that includes both real and synthetic samples, manually annotated to identify personally identifiable information (PII) such as names, dates, locations, and healthcare identifiers. To benchmark our dataset and assess anonymization performance, we evaluate two anonymization strategies: (i) an extractive strategy based on Named Entity Recognition (NER) using BERT-based models, and (ii) a generative strategy using T5-based and GPT-based models to rewrite texts while masking sensitive entities. We conduct a comprehensive series of experiments to evaluate and compare anonymization strategies. Specifically, we assess the impact of incorporating synthetic generated records on model performance by contrasting models fine-tuned solely on real data with those fine-tuned on synthetic samples. We also investigate whether pre-training on biomedical corpora or task-specific fine-tuning more effectively improves performance in the anonymization task. Finally, to support robust evaluation, we introduce an LLM-as-a-Judge framework that leverages a reasoning Large Language Model (LLM) to score anonymization quality, estimate information loss, and assess reidentification risk. Model performance was primarily evaluated using the F1 score on a held-out test set.

All evaluated models achieved good performance in the anonymization task, with the best models reaching F1 scores above 0.90. Both extractive and generative approaches were effective in identifying and masking sensitive entities while preserving the clinical meaning of the texts. Experiments also revealed that including synthetic data improved model generalization, and that task-specific fine-tuning yielded greater performance gains than pre-training the model on biomedical domain.

To the best of our knowledge, AnonyMed-BR is the first manually annotated anonymization dataset for Brazilian Portuguese medical texts, enabling systematic evaluation of both extractive and generative models. The dataset and methodology establish a foundation for privacy-preserving NLP research in the Brazilian healthcare context and the good performance achieved by all models demonstrates the feasibility of developing reliable anonymization systems for Brazilian clinical data. Importantly, the ability to anonymize sensitive information opens opportunities to create new datasets and train models for a variety of downstream tasks in the medical domain, such as clinical outcome prediction, medical entity recognition, diagnostic support, and patient stratification, fostering the growth of NLP research for Brazilian Portuguese healthcare texts. Motivated by our findings, future work includes a deeper exploration of synthetic data generation and utilization. Additionally, we plan to evaluate the models across different languages and textual domains, and to expand the dataset to cover these new languages and domains. These efforts aim to develop more complex anonymization systems with higher generalization capability, ultimately enabling broader applications and safer sharing of data in diverse research and operational settings. All resources are publicly available at https://github.com/venturusbr/AnonyMED-BR.

Figures