Which Samples Should be Learned First: Easy or Hard?

TL;DR

This paper investigates whether easy, hard, or other samples should be learned first in training, providing theoretical analysis and experiments to determine optimal sample prioritization modes based on difficulty distribution.

Contribution

It introduces a unified framework with an optimized objective function and flexible mode switching to select the best sample learning order without prior knowledge.

Findings

The optimal sample learning order depends on difficulty distribution.

Four priority modes are identified: easy, medium, hard, and two-ends.

Flexible switching among modes improves training effectiveness.

Abstract

An effective weighting scheme for training samples is essential for learning tasks. Numerous weighting schemes have been proposed. Some schemes take the easy-first mode, whereas some others take the hard-first one. Naturally, an interesting yet realistic question is raised. Which samples should be learned first given a new learning task, easy or hard? To answer this question, both theoretical analyses and experimental verification are conducted. First, a general optimized objective function is proposed, revealing the relationship between the difficulty distribution and the difficulty-based sample weights. Second, on the basis of the optimized objective function, theoretical answers are obtained. Besides the easy-first and hard-first modes, there are two other priority modes, namely, medium-first and two-ends-first. The prior mode does not necessarily remain unchanged during the training process. Third, an effective and universal solution is proposed to select the optimal priority mode when there is no prior knowledge or theoretical clues. The four modes, namely, easy/medium/hard/two-ends-first, can be flexibly switched in the proposed solution. Fourth, a wide range of experiments is conducted under various scenarios to further compare the weighting schemes in different modes. On the basis of these works, reasonable and comprehensive answers are obtained. Factors including the distribution of samples' learning difficulties and the validation data determine which samples should be learned first in a learning task.