Longitudinal assessment of demographic representativeness in the Medical Imaging and Data Resource Center Open Data Commons

TL;DR

This study evaluates how well the demographic makeup of the MIDRC open medical imaging dataset reflects the US population and COVID-19 cases over time, highlighting the importance of representativeness for fair AI development.

Contribution

It introduces a longitudinal assessment method using Jensen Shannon distance to measure demographic representativeness in open medical imaging data.

Findings

Demographic representativeness varies by ethnicity and race due to data reporting gaps.

Sex and race distributions have remained stable over time.

Metrics like JSD are useful for tracking dataset representativeness.

Abstract

Purpose: The Medical Imaging and Data Resource Center (MIDRC) open data commons was launched to accelerate the development of artificial intelligence (AI) algorithms to help address the COVID-19 pandemic. The purpose of this study was to quantify longitudinal representativeness of the demographic characteristics of the primary imaging dataset compared to the United States general population (US Census) and COVID-19 positive case counts from the Centers for Disease Control and Prevention (CDC). Approach: The Jensen Shannon distance (JSD) was used to longitudinally measure the similarity of the distribution of (1) all unique patients in the MIDRC data to the 2020 US Census and (2) all unique COVID-19 positive patients in the MIDRC data to the case counts reported by the CDC. The distributions were evaluated in the demographic categories of age at index, sex, race, ethnicity, and the intersection of race and ethnicity. Results: Representativeness the MIDRC data by ethnicity and the intersection of race and ethnicity was impacted by the percentage of CDC case counts for which data in these categories is not reported. The distributions by sex and race have retained their level of representativeness over time. Conclusion: The representativeness of the open medical imaging datasets in the curated public data commons at MIDRC has evolved over time as both the number of contributing institutions and overall number of subjects has grown. The use of metrics such as the JSD support measurement of representativeness, one step needed for fair and generalizable AI algorithm development.