A Comparative Study of Light-weight Language Models for PII Masking and their Deployment for Real Conversational Texts

TL;DR

This study compares lightweight encoder-decoder and decoder-only models for PII masking in conversational texts, showing they can perform comparably to large models with benefits in efficiency and control, suitable for real-time applications.

Contribution

It demonstrates that lightweight models like T5-small and Mistral can match large models in PII masking performance, highlighting their advantages in efficiency and controllability for deployment.

Findings

Lightweight models achieve comparable PII masking performance to large models.

Label normalization improves PII masking accuracy across models.

Mistral has higher recall but greater latency; T5 offers better control and lower cost.

Abstract

Automated masking of Personally Identifiable Information (PII) is critical for privacy-preserving conversational systems. While current frontier large language models demonstrate strong PII masking capabilities, concerns about data handling and computational costs motivate exploration of whether lightweight models can achieve comparable performance. We compare encoder-decoder and decoder-only architectures by fine-tuning T5-small and Mistral-Instruct-v0.3 on English datasets constructed from the AI4Privacy benchmark. We create different dataset variants to study label standardization and PII representation, covering 24 standardized PII categories and higher-granularity settings. Evaluation using entity-level and character-level metrics, type accuracy, and exact match shows that both lightweight models achieve performance comparable to frontier LLMs for PII masking tasks. Label normalization consistently improves performance across architectures. Mistral achieves higher F1 and recall with greater robustness across PII types but incurs significantly higher generation latency. T5, while less robust in conversational text, offers more controllable structured outputs and lower inference cost, motivating its use in a real-time Discord bot for real-world PII redaction. Evaluation on live messages reveals performance degradation under informal inputs. These results clarify trade-offs between accuracy, robustness, and computational efficiency, demonstrating that lightweight models can provide effective PII masking while addressing data handling concerns associated with frontier LLMs.