Learning from Subjective Ratings Using Auto-Decoded Deep Latent Embeddings

TL;DR

This paper introduces ADDLE, a novel deep learning framework that explicitly models rater subjectivity in medical diagnosis, improving accuracy in liver steatosis detection from ultrasound data despite label variability.

Contribution

ADDLE is the first method to explicitly incorporate rater tendencies using auto-decoded latent embeddings, enabling learning from noisy, subjective labels without increasing complexity with more raters.

Findings

ADDLE improves AUC for severe steatosis diagnosis by 10.5%.

It outperforms other noise-robust methods with fewer parameters.

The approach effectively models rater variability in large-scale medical data.

Abstract

Depending on the application, radiological diagnoses can be associated with high inter- and intra-rater variabilities. Most computer-aided diagnosis (CAD) solutions treat such data as incontrovertible, exposing learning algorithms to considerable and possibly contradictory label noise and biases. Thus, managing subjectivity in labels is a fundamental problem in medical imaging analysis. To address this challenge, we introduce auto-decoded deep latent embeddings (ADDLE), which explicitly models the tendencies of each rater using an auto-decoder framework. After a simple linear transformation, the latent variables can be injected into any backbone at any and multiple points, allowing the model to account for rater-specific effects on the diagnosis. Importantly, ADDLE does not expect multiple raters per image in training, meaning it can readily learn from data mined from hospital archives. Moreover, the complexity of training ADDLE does not increase as more raters are added. During inference each rater can be simulated and a 'mean' or 'greedy' virtual rating can be produced. We test ADDLE on the problem of liver steatosis diagnosis from 2D ultrasound (US) by collecting 46 084 studies along with clinical US diagnoses originating from 65 different raters. We evaluated diagnostic performance using a separate dataset with gold-standard biopsy diagnoses. ADDLE can improve the partial areas under the curve (AUCs) for diagnosing severe steatosis by 10.5% over standard classifiers while outperforming other annotator-noise approaches, including those requiring 65 times the parameters.