A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning

TL;DR

This paper presents WikiPII, a large automatically labeled dataset for personal information extraction, and demonstrates how federated learning can effectively train NER models in privacy-sensitive environments despite noisy labels.

Contribution

It introduces WikiPII, a novel dataset for personal information extraction, and explores federated learning to train models on noisy, privacy-sensitive data.

Findings

Large dataset improves model accuracy despite label noise

Federated learning is effective for privacy-preserving training

Automatic annotation reduces manual labeling costs

Abstract

We curated WikiPII, an automatically labeled dataset composed of Wikipedia biography pages, annotated for personal information extraction. Although automatic annotation can lead to a high degree of label noise, it is an inexpensive process and can generate large volumes of annotated documents. We trained a BERT-based NER model with WikiPII and showed that with an adequately large training dataset, the model can significantly decrease the cost of manual information extraction, despite the high level of label noise. In a similar approach, organizations can leverage text mining techniques to create customized annotated datasets from their historical data without sharing the raw data for human annotation. Also, we explore collaborative training of NER models through federated learning when the annotation is noisy. Our results suggest that depending on the level of trust to the ML operator and the volume of the available data, distributed training can be an effective way of training a personal information identifier in a privacy-preserved manner. Research material is available at https://github.com/ratmcu/wikipiifed.