# Cross-Modal Data Programming Enables Rapid Medical Machine Learning

## TL;DR

Cross-modal data programming allows rapid, clinician-driven labeling of medical data by writing rules over auxiliary modalities, significantly reducing labeling time while maintaining high model performance.

## Contribution

We introduce a theoretically-grounded cross-modal data programming method that simplifies clinician input, reduces labeling time, and leverages unlabeled data for medical machine learning.

## Key findings

- Achieves comparable or better performance than extensive manual labeling
- Reduces labeling time from months to hours
- Effective across multiple medical modalities

## Abstract

Labeling training datasets has become a key barrier to building medical machine learning models. One strategy is to generate training labels programmatically, for example by applying natural language processing pipelines to text reports associated with imaging studies. We propose cross-modal data programming, which generalizes this intuitive strategy in a theoretically-grounded way that enables simpler, clinician-driven input, reduces required labeling time, and improves with additional unlabeled data. In this approach, clinicians generate training labels for models defined over a target modality (e.g. images or time series) by writing rules over an auxiliary modality (e.g. text reports). The resulting technical challenge consists of estimating the accuracies and correlations of these rules; we extend a recent unsupervised generative modeling technique to handle this cross-modal setting in a provably consistent way. Across four applications in radiography, computed tomography, and electroencephalography, and using only several hours of clinician time, our approach matches or exceeds the efficacy of physician-months of hand-labeling with statistical significance, demonstrating a fundamentally faster and more flexible way of building machine learning models in medicine.