Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization

TL;DR

Tau-Eval is an open-source framework designed to comprehensively evaluate text anonymization methods by balancing privacy protection and utility across diverse applications.

Contribution

It introduces a unified benchmarking framework for text anonymization that considers both privacy and utility, addressing the lack of universal evaluation standards.

Findings

Provides a standardized way to compare anonymization techniques

Balances privacy and utility in evaluation metrics

Supports diverse downstream tasks for comprehensive assessment

Abstract

Text anonymization is the process of removing or obfuscating information from textual data to protect the privacy of individuals. This process inherently involves a complex trade-off between privacy protection and information preservation, where stringent anonymization methods can significantly impact the text's utility for downstream applications. Evaluating the effectiveness of text anonymization proves challenging from both privacy and utility perspectives, as there is no universal benchmark that can comprehensively assess anonymization techniques across diverse, and sometimes contradictory contexts. We present Tau-Eval, an open-source framework for benchmarking text anonymization methods through the lens of privacy and utility task sensitivity. A Python library, code, documentation and tutorials are publicly available.