CONAN: Complementary Pattern Augmentation for Rare Disease Detection

TL;DR

CONAN is a novel framework that leverages adversarial training and max-margin classification to improve detection of rare diseases by generating and utilizing uncertain patient data.

Contribution

It introduces a complementary GAN-based augmentation method combined with self-attentive embeddings for rare disease detection, addressing low prevalence and uncertain diagnoses.

Findings

Achieved 0.96 PR-AUC in IBD detection with 50.1% improvement.

Achieved 0.22 PR-AUC in IPF detection with 41.3% improvement.

Demonstrated effectiveness on two rare disease detection tasks.

Abstract

Rare diseases affect hundreds of millions of people worldwide but are hard to detect since they have extremely low prevalence rates (varying from 1/1,000 to 1/200,000 patients) and are massively underdiagnosed. How do we reliably detect rare diseases with such low prevalence rates? How to further leverage patients with possibly uncertain diagnosis to improve detection? In this paper, we propose a Complementary pattern Augmentation (CONAN) framework for rare disease detection. CONAN combines ideas from both adversarial training and max-margin classification. It first learns self-attentive and hierarchical embedding for patient pattern characterization. Then, we develop a complementary generative adversarial networks (GAN) model to generate candidate positive and negative samples from the uncertain patients by encouraging a max-margin between classes. In addition, CONAN has a disease detector that serves as the discriminator during the adversarial training for identifying rare diseases. We evaluated CONAN on two disease detection tasks. For low prevalence inflammatory bowel disease (IBD) detection, CONAN achieved .96 precision recall area under the curve (PR-AUC) and 50.1% relative improvement over best baseline. For rare disease idiopathic pulmonary fibrosis (IPF) detection, CONAN achieves .22 PR-AUC with 41.3% relative improvement over the best baseline.