An ELIXIR scoping review on domain-specific evaluation metrics for synthetic data in life sciences

TL;DR

This paper reviews current evaluation metrics for synthetic data in life sciences, highlighting the need for standardized, robust methods to improve trust, validation, and application across various domains.

Contribution

It systematically analyzes existing practices for assessing synthetic data in life sciences and emphasizes the necessity for standardized evaluation metrics.

Findings

Evaluation practices are often inconsistent across domains.

Systematic evaluation of synthetic data is frequently overlooked.

Standardized metrics are urgently needed for better validation.

Abstract

Synthetic data has emerged as a powerful resource in life sciences, offering solutions for data scarcity, privacy protection and accessibility constraints. By creating artificial datasets that mirror the characteristics of real data, allows researchers to develop and validate computational methods in controlled environments. Despite its promise, the adoption of synthetic data in Life Sciences hinges on rigorous evaluation metrics designed to assess their fidelity and reliability. To explore the current landscape of synthetic data evaluation metrics in several Life Sciences domains, the ELIXIR Machine Learning Focus Group performed a systematic review of the scientific literature following the PRISMA guidelines. Six critical domains were examined to identify current practices for assessing synthetic data. Findings reveal that, while generation methods are rapidly evolving, systematic evaluation is often overlooked, limiting researchers ability to compare, validate, and trust synthetic datasets across different domains. This systematic review underscores the urgent need for robust, standardized evaluation approaches that not only bolster confidence in synthetic data but also guide its effective and responsible implementation. By laying the groundwork for establishing domain-specific yet interoperable standards, this scoping review paves the way for future initiatives aimed at enhancing the role of synthetic data in scientific discovery, clinical practice and beyond.