Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting

TL;DR

Meta-Weight-Net introduces an adaptive, learnable sample weighting function modeled by an MLP, which improves deep neural network training on biased data by automatically fitting optimal weights guided by meta-data.

Contribution

It proposes a novel method to learn an explicit weighting function directly from data, eliminating manual design and hyper-parameter tuning of sample weights.

Findings

Achieves better accuracy than state-of-the-art methods.

Effectively handles class imbalance and noisy labels.

Adapts to complex scenarios beyond traditional settings.

Abstract

Current deep neural networks (DNNs) can easily overfit to biased training data with corrupted labels or class imbalance. Sample re-weighting strategy is commonly used to alleviate this issue by designing a weighting function mapping from training loss to sample weight, and then iterating between weight recalculating and classifier updating. Current approaches, however, need manually pre-specify the weighting function as well as its additional hyper-parameters. It makes them fairly hard to be generally applied in practice due to the significant variation of proper weighting schemes relying on the investigated problem and training data. To address this issue, we propose a method capable of adaptively learning an explicit weighting function directly from data. The weighting function is an MLP with one hidden layer, constituting a universal approximator to almost any continuous functions, making the method able to fit a wide range of weighting functions including those assumed in conventional research. Guided by a small amount of unbiased meta-data, the parameters of the weighting function can be finely updated simultaneously with the learning process of the classifiers. Synthetic and real experiments substantiate the capability of our method for achieving proper weighting functions in class imbalance and noisy label cases, fully complying with the common settings in traditional methods, and more complicated scenarios beyond conventional cases. This naturally leads to its better accuracy than other state-of-the-art methods.