Federated pretraining and fine tuning of BERT using clinical notes from multiple silos

TL;DR

This paper demonstrates that BERT can be pretrained and fine-tuned in a federated setting using clinical notes from multiple institutions, preserving privacy while enabling large-scale healthcare NLP.

Contribution

It introduces a federated approach for pretraining and fine-tuning BERT on clinical data across multiple silos without data sharing.

Findings

Successful federated pretraining of BERT on clinical notes

Effective federated fine-tuning for healthcare NLP tasks

Preservation of data privacy during model training

Abstract

Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.