Utility-Preserving De-Identification for Math Tutoring: Investigating Numeric Ambiguity in the MathEd-PII Benchmark Dataset

TL;DR

This paper introduces MathEd-PII, a benchmark dataset for PII detection in math tutoring dialogues, and demonstrates that domain-aware, segment-aware prompting significantly improves de-identification accuracy while maintaining educational utility.

Contribution

The work presents the first benchmark dataset for PII detection in math tutoring data and shows that domain-aware prompting enhances de-identification performance.

Findings

Domain-aware prompting achieves F1: 0.802 and 0.821, outperforming baseline.

False PII redactions cluster in math-dense regions, indicating numeric ambiguity.

Domain context is essential for utility-preserving de-identification.

Abstract

Large-scale sharing of dialogue data is key to advancing the science of teaching and learning, yet rigorous de-identification remains a major barrier. In mathematics tutoring transcripts, numeric expressions frequently resemble structured identifiers (e.g., dates or IDs), leading generic Personally Identifiable Information (PII) detection systems to over-redact core instructional content and reduce data utility. This work asks how to detect PII while preserving educational utility, focusing on this "numeric ambiguity" problem. We introduce MathEd-PII, the first benchmark dataset for PII detection in math tutoring dialogues, built with human-in-the-loop LLM annotation. Using density-based segmentation, we show that false PII redactions cluster in math-dense regions, confirming numeric ambiguity as a key failure mode. We then compare four detection strategies: a Presidio baseline and three LLM-based approaches with basic, math-aware, and segment-aware prompting. Domain-aware prompting, including both math-aware (F1: 0.802) and segment-aware versions (F1: 0.821), substantially outperforms the baseline (F1: 0.379) while reducing numeric false positives, demonstrating that de-identification must incorporate domain context to preserve analytic utility. This work provides a new benchmark and evidence that utility-preserving de-identification for tutoring data requires domain-aware modeling.