A Study on the Predictability of Sample Learning Consistency

TL;DR

This paper investigates the predictability of sample difficulty in curriculum learning using C-Score, revealing challenges in generalization and suggesting that sample relations may better explain difficulty measures.

Contribution

It evaluates models predicting C-Score for CIFAR datasets and highlights the limitations of current approaches, proposing sample relations as a future direction.

Findings

Models poorly generalize within and out of distribution.

C-Score is influenced by factors beyond individual sample characteristics.

Sample relations, such as label sharing among neighbors, may better explain difficulty.

Abstract

Curriculum Learning is a powerful training method that allows for faster and better training in some settings. This method, however, requires having a notion of which examples are difficult and which are easy, which is not always trivial to provide. A recent metric called C-Score acts as a proxy for example difficulty by relating it to learning consistency. Unfortunately, this method is quite compute intensive which limits its applicability for alternative datasets. In this work, we train models through different methods to predict C-Score for CIFAR-100 and CIFAR-10. We find, however, that these models generalize poorly both within the same distribution as well as out of distribution. This suggests that C-Score is not defined by the individual characteristics of each sample but rather by other factors. We hypothesize that a sample's relation to its neighbours, in particular, how many of them share the same labels, can help in explaining C-Scores. We plan to explore this in future work.