Robust Neural Regression via Uncertainty Learning

TL;DR

This paper introduces a simple, robust neural regression method that estimates uncertainty effectively using a dual-network approach, avoiding complex training procedures and auxiliary data required by previous methods.

Contribution

It extends iterative reweighted least squares with two sub-networks for prediction and uncertainty, providing a simpler and more robust alternative to existing uncertainty estimation techniques.

Findings

Outperforms complex models like MC Dropout and SDE-Net in robustness.

Easier to implement with shared representations and cooperative training.

Insensitive to varying types of uncertainty.

Abstract

Deep neural networks tend to underestimate uncertainty and produce overly confident predictions. Recently proposed solutions, such as MC Dropout and SDENet, require complex training and/or auxiliary out-of-distribution data. We propose a simple solution by extending the time-tested iterative reweighted least square (IRLS) in generalised linear regression. We use two sub-networks to parametrise the prediction and uncertainty estimation, enabling easy handling of complex inputs and nonlinear response. The two sub-networks have shared representations and are trained via two complementary loss functions for the prediction and the uncertainty estimates, with interleaving steps as in a cooperative game. Compared with more complex models such as MC-Dropout or SDE-Net, our proposed network is simpler to implement and more robust (insensitive to varying aleatoric and epistemic uncertainty).