PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection

TL;DR

PIIBench is a comprehensive, unified benchmark corpus for PII detection, consolidating multiple datasets and providing a challenging evaluation platform for existing systems.

Contribution

The paper introduces PIIBench, a large, standardized PII dataset with a normalization pipeline and baseline evaluations, addressing fragmentation and difficulty in PII detection benchmarks.

Findings

All evaluated systems achieved span-level F1 below 0.14.

The best system (Presidio) had an F1 of 0.1385 and zero recall on most entity types.

PIIBench is more challenging than existing single-source PII datasets.

Abstract

We present PIIBench, a unified benchmark corpus for Personally Identifiable Information (PII) detection in natural language text. Existing resources for PII detection are fragmented across domain-specific corpora with mutually incompatible annotation schemes, preventing systematic comparison of detection systems. We consolidate ten publicly available datasets spanning synthetic PII corpora, multilingual Named Entity Recognition (NER) benchmarks, and financial domain annotated text, yielding a corpus of 2,369,883 annotated sequences and 3.35 million entity mentions across 48 canonical PII entity types. We develop a principled normalization pipeline that maps 80+ source-specific label variants to a standardized BIO tagging scheme, applies frequency-based suppression of near absent entity types, and produces stratified 80/10/10 train/validation/test splits preserving source distribution. To establish baseline difficulty, we evaluate eight published systems spanning rule-based engines (Microsoft Presidio), general purpose NER models (spaCy, BERT-base NER, XLM-RoBERTa NER, SpanMarker mBERT, SpanMarker BERT), a PII-specific model (Piiranha DeBERTa), and a financial NER specialist (XtremeDistil FiNER). All systems achieve span-level F1 below 0.14, with the best system (Presidio, F1=0.1385) still producing zero recall on most entity types. These results directly quantify the domain-silo problem and demonstrate that PIIBench presents a substantially harder and more comprehensive evaluation challenge than any existing single source PII dataset. The dataset construction pipeline and benchmark evaluation code are publicly available at https://github.com/pritesh-2711/pii-bench.