An ELIXIR scoping review on domain-specific evaluation metrics for synthetic data in life sciences

TL;DR

This paper reviews how synthetic data is evaluated in life sciences and highlights the need for standardized metrics to ensure reliability and trustworthiness.

Contribution

The paper provides a systematic review of evaluation metrics for synthetic data across six life science domains, revealing gaps in current practices.

Findings

Synthetic data generation methods are evolving rapidly, but systematic evaluation is often neglected.

Current evaluation practices limit the ability to compare and trust synthetic datasets across domains.

There is a clear need for robust, standardized metrics to guide the responsible use of synthetic data.

Abstract

Synthetic data (SD) has become an increasingly important asset in the life sciences, helping address data scarcity, privacy concerns, and barriers to data access. Creating artificial datasets that mirror the characteristics of real data allows researchers to develop and validate computational methods in controlled environments. Despite its promise, the adoption of SD in life sciences hinges on rigorous evaluation metrics designed to assess their fidelity and reliability. To explore the current landscape of SD evaluation metrics in distinct life sciences domains, the ELIXIR Machine Learning Focus Group performed a systematic review of the scientific literature following the PRISMA guidelines. Six critical domains were examined to identify current practices for assessing SD. Findings reveal that, while generation methods are rapidly evolving, systematic evaluation is often overlooked, limiting researchers’ ability to compare, validate, and trust synthetic datasets across different domains. This systematic review underscores the urgent need for robust, standardized evaluation approaches that not only bolster confidence in SD but also guide its effective and responsible implementation. By laying the groundwork for establishing domain-specific yet interoperable standards, this scoping review paves the way for future initiatives aimed at enhancing the role of SD in scientific discovery, clinical practice and beyond.

Figures