# Robustness Against Outliers For Deep Neural Networks By Gradient Conjugate Priors

## TL;DR

This paper introduces a gradient conjugate prior (GCP) network that robustly estimates probability distributions in the presence of outliers, providing explicit bias correction formulas and demonstrating superior performance over existing methods.

## Contribution

The paper develops a novel GCP network with explicit bias correction formulas for outlier-affected data, improving robustness in distribution reconstruction.

## Key findings

- GCP network effectively corrects bias caused by outliers.
- Fitted mean is close to ground truth within an exponential neighborhood.
- Corrected variance remains close to true variance, even with high outlier proportion.

## Abstract

We analyze a new robust method for the reconstruction of probability distributions of observed data in the presence of output outliers. It is based on a so-called gradient conjugate prior (GCP) network which outputs the parameters of a prior. By rigorously studying the dynamics of the GCP learning process, we derive an explicit formula for correcting the obtained variance of the marginal distribution and removing the bias caused by outliers in the training set. Assuming a Gaussian (input-dependent) ground truth distribution contaminated with a proportion $\varepsilon$ of outliers, we show that the fitted mean is in a $c e^{-1/\varepsilon}$-neighborhood of the ground truth mean and the corrected variance is in a $b\varepsilon$-neighborhood of the ground truth variance, whereas the uncorrected variance of the marginal distribution can even be infinite. We explicitly find $b$ as a function of the output of the GCP network, without a priori knowledge of the outliers (possibly input-dependent) distribution. Experiments with synthetic and real-world data sets indicate that the GCP network fitted with a standard optimizer outperforms other robust methods for regression.