Human not in the loop: objective sample difficulty measures for Curriculum Learning

TL;DR

This paper introduces an automated, objective difficulty measure called VoG for curriculum learning in medical image classification, eliminating the need for subjective human annotations and improving model performance.

Contribution

The work proposes a novel automated difficulty metric using gradient variance, enabling objective curriculum learning without human bias in medical image classification.

Findings

VoG-based curriculum improves classification accuracy

Comparable or better results than human-annotated difficulty measures

Effective in binary and multi-class fracture classification

Abstract

Curriculum learning is a learning method that trains models in a meaningful order from easier to harder samples. A key here is to devise automatic and objective difficulty measures of samples. In the medical domain, previous work applied domain knowledge from human experts to qualitatively assess classification difficulty of medical images to guide curriculum learning, which requires extra annotation efforts, relies on subjective human experience, and may introduce bias. In this work, we propose a new automated curriculum learning technique using the variance of gradients (VoG) to compute an objective difficulty measure of samples and evaluated its effects on elbow fracture classification from X-ray images. Specifically, we used VoG as a metric to rank each sample in terms of the classification difficulty, where high VoG scores indicate more difficult cases for classification, to guide the curriculum training process We compared the proposed technique to a baseline (without curriculum learning), a previous method that used human annotations on classification difficulty, and anti-curriculum learning. Our experiment results showed comparable and higher performance for the binary and multi-class bone fracture classification tasks.