Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression

TL;DR

This paper introduces Batch Inverse-Variance, a robust loss function for heteroscedastic regression that leverages per-label noise variance estimates to improve neural network performance on noisy datasets.

Contribution

The paper proposes a novel Batch Inverse-Variance loss function that enhances heteroscedastic regression by effectively handling noisy labels and controlling learning rates.

Findings

BIV significantly outperforms L2 loss and inverse-variance weighting on noisy datasets.

The method is robust to near-ground truth samples.

Experimental results demonstrate improved accuracy over baseline methods.

Abstract

Heteroscedastic regression is the task of supervised learning where each label is subject to noise from a different distribution. This noise can be caused by the labelling process, and impacts negatively the performance of the learning algorithm as it violates the i.i.d. assumptions. In many situations however, the labelling process is able to estimate the variance of such distribution for each label, which can be used as an additional information to mitigate this impact. We adapt an inverse-variance weighted mean square error, based on the Gauss-Markov theorem, for parameter optimization on neural networks. We introduce Batch Inverse-Variance, a loss function which is robust to near-ground truth samples, and allows to control the effective learning rate. Our experimental results show that BIV improves significantly the performance of the networks on two noisy datasets, compared to L2 loss, inverse-variance weighting, as well as a filtering-based baseline.