FEST: A Unified Framework for Evaluating Synthetic Tabular Data

TL;DR

FEST is a comprehensive, open-source framework for evaluating synthetic tabular data, balancing privacy protection and data utility through diverse metrics, and validated across multiple datasets.

Contribution

This paper introduces FEST, a unified evaluation framework that systematically assesses privacy and utility in synthetic tabular data generation.

Findings

FEST effectively analyzes privacy-utility trade-offs.

It integrates multiple privacy and utility metrics.

Validated on diverse datasets, demonstrating robustness.

Abstract

Synthetic data generation, leveraging generative machine learning techniques, offers a promising approach to mitigating privacy concerns associated with real-world data usage. Synthetic data closely resembles real-world data while maintaining strong privacy guarantees. However, a comprehensive assessment framework is still missing in the evaluation of synthetic data generation, especially when considering the balance between privacy preservation and data utility in synthetic data. This research bridges this gap by proposing FEST, a systematic framework for evaluating synthetic tabular data. FEST integrates diverse privacy metrics (attack-based and distance-based), along with similarity and machine learning utility metrics, to provide a holistic assessment. We develop FEST as an open-source Python-based library and validate it on multiple datasets, demonstrating its effectiveness in analyzing the privacy-utility trade-off of different synthetic data generation models. The source code of FEST is available on Github.