Was that so hard? Estimating human classification difficulty

TL;DR

This paper presents methods to automatically estimate the difficulty of medical diagnosis cases using deep metric learning, improving over existing approaches and aiding training strategies for doctors.

Contribution

It introduces novel techniques for estimating diagnostic difficulty from medical images, utilizing deep embeddings and self-assessed certainty, applicable with or without ground truth difficulty labels.

Findings

High correlation with human difficulty assessments

Outperforms existing difficulty estimation methods

Effective on multiple medical datasets

Abstract

When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients, showing that we outperform existing methods by a large margin on our problem and data.