ICD Codes are Insufficient to Create Datasets for Machine Learning: An Evaluation Using All of Us Data for Coccidioidomycosis and Myocardial Infarction

TL;DR

This study evaluates the reliability of ICD codes for creating ML datasets in medicine by comparing them with serological and laboratory-confirmed patient cohorts for coccidioidomycosis and myocardial infarction, revealing significant discrepancies.

Contribution

It demonstrates that ICD codes alone are insufficient for accurately constructing ML training datasets in medical research, highlighting the need for more precise data collection methods.

Findings

Significant discrepancies between ICD-based and confirmed diagnosis cohorts.

Small overlap between ICD-coded and laboratory-confirmed patient groups.

Variations in demographics and clinical data across cohorts.

Abstract

In medicine, machine learning (ML) datasets are often built using the International Classification of Diseases (ICD) codes. As new models are being developed, there is a need for larger datasets. However, ICD codes are intended for billing. We aim to determine how suitable ICD codes are for creating datasets to train ML models. We focused on a rare and common disease using the All of Us database. First, we compared the patient cohort created using ICD codes for Valley fever (coccidioidomycosis, CM) with that identified via serological confirmation. Second, we compared two similarly created patient cohorts for myocardial infarction (MI) patients. We identified significant discrepancies between these two groups, and the patient overlap was small. The CM cohort had 811 patients in the ICD-10 group, 619 patients in the positive-serology group, and 24 with both. The MI cohort had 14,875 patients in the ICD-10 group, 23,598 in the MI laboratory-confirmed group, and 6,531 in both. Demographics, rates of disease symptoms, and other clinical data varied across our case study cohorts.