Downstream Fairness Caveats with Synthetic Healthcare Data

TL;DR

This study assesses the fairness of synthetic healthcare data generated by GANs, revealing that synthetic data exhibits different bias characteristics than real data and that fairness mitigation techniques have varied effects.

Contribution

It provides a systematic evaluation of biases in synthetic healthcare data and compares the impact of fairness mitigation methods on data utility and fairness.

Findings

Synthetic data shows different fairness properties than real data.

Fairness mitigation techniques have varied effects on synthetic data.

Synthetic data is not inherently bias-free.

Abstract

This paper evaluates synthetically generated healthcare data for biases and investigates the effect of fairness mitigation techniques on utility-fairness. Privacy laws limit access to health data such as Electronic Medical Records (EMRs) to preserve patient privacy. Albeit essential, these laws hinder research reproducibility. Synthetic data is a viable solution that can enable access to data similar to real healthcare data without privacy risks. Healthcare datasets may have biases in which certain protected groups might experience worse outcomes than others. With the real data having biases, the fairness of synthetically generated health data comes into question. In this paper, we evaluate the fairness of models generated on two healthcare datasets for gender and race biases. We generate synthetic versions of the dataset using a Generative Adversarial Network called HealthGAN, and compare the real and synthetic model's balanced accuracy and fairness scores. We find that synthetic data has different fairness properties compared to real data and fairness mitigation techniques perform differently, highlighting that synthetic data is not bias free.