Jump-teaching: Combating Sample Selection Bias via Temporal Disagreement

TL;DR

Jump-teaching is an efficient method that leverages temporal disagreement within a single neural network to combat sample selection bias, improving robustness and training efficiency without extra overhead.

Contribution

It introduces a novel jump-manner update strategy and a fine-grained, single-network based selection criterion to effectively address sample selection bias.

Findings

Outperforms state-of-the-art methods in robustness.

Speeds up training by up to 4.47 times.

Reduces peak memory footprint by 54%.

Abstract

Sample selection is a straightforward technique to combat noisy labels, aiming to prevent mislabeled samples from degrading the robustness of neural networks. However, existing methods mitigate compounding selection bias either by leveraging dual-network disagreement or additional forward propagations, leading to multiplied training overhead. To address this challenge, we introduce $\textit{Jump-teaching}$, an efficient sample selection framework for debiased model update and simplified selection criterion. Based on a key observation that a neural network exhibits significant disagreement across different training iterations, Jump-teaching proposes a jump-manner model update strategy to enable self-correction of selection bias by harnessing temporal disagreement, eliminating the need for multi-network or multi-round training. Furthermore, we employ a sample-wise selection criterion building on the intra variance of a decomposed single loss for a fine-grained selection without relying on batch-wise ranking or dataset-wise modeling. Extensive experiments demonstrate that Jump-teaching outperforms state-of-the-art counterparts while achieving a nearly overhead-free selection procedure, which boosts training speed by up to $4.47\times$ and reduces peak memory footprint by $54\%$.