Detection of Personal Data in Structured Datasets Using a Large Language Model

TL;DR

This paper introduces a new method using GPT-4o to detect personal data in structured datasets by incorporating contextual information, showing improved performance especially on real-world datasets compared to existing tools.

Contribution

The paper presents a novel approach leveraging GPT-4o with contextual information for personal data detection, outperforming existing methods on real-world datasets.

Findings

GPT-4o-based approach outperforms others on MIMIC-Demo-Ext

Contextual information improves detection in Kaggle and OpenML datasets

Performance varies significantly across datasets

Abstract

We propose a novel approach for detecting personal data in structured datasets, leveraging GPT-4o, a state-of-the-art Large Language Model. A key innovation of our method is the incorporation of contextual information: in addition to a feature's name and values, we utilize information from other feature names within the dataset as well as the dataset description. We compare our approach to alternative methods, including Microsoft Presidio and CASSED, evaluating them on multiple datasets: DeSSI, a large synthetic dataset, datasets we collected from Kaggle and OpenML as well as MIMIC-Demo-Ext, a real-world dataset containing patient information from critical care units. Our findings reveal that detection performance varies significantly depending on the dataset used for evaluation. CASSED excels on DeSSI, the dataset on which it was trained. Performance on the medical dataset MIMIC-Demo-Ext is comparable across all models, with our GPT-4o-based approach clearly outperforming the others. Notably, personal data detection in the Kaggle and OpenML datasets appears to benefit from contextual information. This is evidenced by the poor performance of CASSED and Presidio (both of which do not utilize the context of the dataset) compared to the strong results of our GPT-4o-based approach. We conclude that further progress in this field would greatly benefit from the availability of more real-world datasets containing personal information.