The LLM Data Auditor: A Metric-oriented Survey on Quality and Trustworthiness in Evaluating Synthetic Data

TL;DR

This paper introduces the LLM Data Auditor framework, a unified, metric-oriented approach to evaluate the intrinsic quality and trustworthiness of synthetic data generated by LLMs across multiple modalities, highlighting current evaluation gaps.

Contribution

It proposes a comprehensive evaluation framework categorizing intrinsic metrics for synthetic data, analyzing existing methods, and providing recommendations for improved data quality assessment across modalities.

Findings

Current evaluation practices have significant deficiencies.

Intrinsic metrics can effectively assess data quality and trustworthiness.

The framework guides practical application of synthetic data across modalities.

Abstract

Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities. By transforming data from a scarce resource into a controllable asset, LLMs mitigate the bottlenecks imposed by the acquisition costs of real-world data for model training, evaluation, and system iteration. However, ensuring the high quality of LLM-generated synthetic data remains a critical challenge. Existing research primarily focuses on generation methodologies, with limited direct attention to the quality of the resulting data. Furthermore, most studies are restricted to single modalities, lacking a unified perspective across different data types. To bridge this gap, we propose the \textbf{LLM Data Auditor framework}. In this framework, we first describe how LLMs are utilized to generate data across six distinct modalities. More importantly, we systematically categorize intrinsic metrics for evaluating synthetic data from two dimensions: quality and trustworthiness. This approach shifts the focus from extrinsic evaluation, which relies on downstream task performance, to the inherent properties of the data itself. Using this evaluation system, we analyze the experimental evaluations of representative generation methods for each modality and identify substantial deficiencies in current evaluation practices. Based on these findings, we offer concrete recommendations for the community to improve the evaluation of data generation. Finally, the framework outlines methodologies for the practical application of synthetic data across different modalities.