Heavy-Tailed Processes for Selective Shrinkage

TL;DR

This paper introduces heavy-tailed stochastic processes derived from Gaussian processes to improve robustness of regression and classification models against outliers in input space, especially in sparse regions, through selective shrinkage.

Contribution

It proposes a novel heavy-tailed process constructed via a copula to enable selective shrinkage in sparse regions, backed by theoretical analysis and biological data experiments.

Findings

Heavy-tailed processes induce selective shrinkage in sparse regions.

Theoretical conditions for effective selective shrinkage are established.

Experiments show improved estimates in sparse regions with competitive dense region performance.

Abstract

Heavy-tailed distributions are frequently used to enhance the robustness of regression and classification methods to outliers in output space. Often, however, we are confronted with "outliers" in input space, which are isolated observations in sparsely populated regions. We show that heavy-tailed stochastic processes (which we construct from Gaussian processes via a copula), can be used to improve robustness of regression and classification estimators to such outliers by selectively shrinking them more strongly in sparse regions than in dense regions. We carry out a theoretical analysis to show that selective shrinkage occurs, provided the marginals of the heavy-tailed process have sufficiently heavy tails. The analysis is complemented by experiments on biological data which indicate significant improvements of estimates in sparse regions while producing competitive results in dense regions.