SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains

TL;DR

SynthTextEval is a comprehensive toolkit designed to evaluate synthetic text data across multiple dimensions, enhancing its utility, fairness, privacy, and domain-specific quality for high-stakes applications like healthcare and law.

Contribution

The paper introduces SynthTextEval, a toolkit that standardizes and consolidates evaluation metrics for synthetic text, addressing multiple critical aspects in high-stakes domains.

Findings

Effective evaluation of synthetic text in healthcare and law domains.

Standardized metrics improve assessment of utility, fairness, and privacy.

Toolkit facilitates better privacy-preserving AI development.

Abstract

We present SynthTextEval, a toolkit for conducting comprehensive evaluations of synthetic text. The fluency of large language model (LLM) outputs has made synthetic text potentially viable for numerous applications, such as reducing the risks of privacy violations in the development and deployment of AI systems in high-stakes domains. Realizing this potential, however, requires principled consistent evaluations of synthetic data across multiple dimensions: its utility in downstream systems, the fairness of these systems, the risk of privacy leakage, general distributional differences from the source text, and qualitative feedback from domain experts. SynthTextEval allows users to conduct evaluations along all of these dimensions over synthetic data that they upload or generate using the toolkit's generation module. While our toolkit can be run over any data, we highlight its functionality and effectiveness over datasets from two high-stakes domains: healthcare and law. By consolidating and standardizing evaluation metrics, we aim to improve the viability of synthetic text, and in-turn, privacy-preservation in AI development.