A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models

TL;DR

This paper introduces SynEval, a comprehensive evaluation framework for assessing the quality, utility, and privacy of synthetic tabular data generated by large language models, validated on product review datasets.

Contribution

The paper presents SynEval, a novel open-source framework that quantitatively evaluates synthetic data from LLMs, addressing the lack of comprehensive assessment tools.

Findings

SynEval effectively measures fidelity, utility, and privacy of synthetic data.

Trade-offs between evaluation metrics are highlighted in synthetic data quality.

SynEval aids researchers in selecting suitable synthetic data for specific applications.

Abstract

The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data,, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.