Dropout can Simulate Exponential Number of Models for Sample Selection Techniques

TL;DR

This paper demonstrates how Dropout can be used to simulate an exponential number of models for sample selection in noisy label training, improving efficiency and effectiveness over traditional two-model methods.

Contribution

It introduces a novel approach to leverage Dropout for training an exponential number of shared models, enhancing sample selection techniques in noisy label scenarios.

Findings

Improved accuracy in noisy label training tasks.

Efficient use of Dropout to simulate multiple models.

Enhanced sample selection performance.

Abstract

Following Coteaching, generally in the literature, two models are used in sample selection based approaches for training with noisy labels. Meanwhile, it is also well known that Dropout when present in a network trains an ensemble of sub-networks. We show how to leverage this property of Dropout to train an exponential number of shared models, by training a single model with Dropout. We show how we can modify existing two model-based sample selection methodologies to use an exponential number of shared models. Not only is it more convenient to use a single model with Dropout, but this approach also combines the natural benefits of Dropout with that of training an exponential number of models, leading to improved results.