Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression

TL;DR

This paper introduces a self-supervised approach for estimating covariance in deep heteroscedastic regression, leveraging a new Wasserstein distance bound and a heuristic for pseudo labels, improving accuracy and efficiency.

Contribution

It proposes a novel self-supervised covariance estimation method using a Wasserstein bound and pseudo labels, reducing computational complexity in deep heteroscedastic regression.

Findings

Effective pseudo labels generated by neighborhood heuristic

Wasserstein bound improves covariance estimation accuracy

Method outperforms unsupervised approaches in experiments

Abstract

Deep heteroscedastic regression models the mean and covariance of the target distribution through neural networks. The challenge arises from heteroscedasticity, which implies that the covariance is sample dependent and is often unknown. Consequently, recent methods learn the covariance through unsupervised frameworks, which unfortunately yield a trade-off between computational complexity and accuracy. While this trade-off could be alleviated through supervision, obtaining labels for the covariance is non-trivial. Here, we study self-supervised covariance estimation in deep heteroscedastic regression. We address two questions: (1) How should we supervise the covariance assuming ground truth is available? (2) How can we obtain pseudo labels in the absence of the ground-truth? We address (1) by analysing two popular measures: the KL Divergence and the 2-Wasserstein distance. Subsequently, we derive an upper bound on the 2-Wasserstein distance between normal distributions with non-commutative covariances that is stable to optimize. We address (2) through a simple neighborhood based heuristic algorithm which results in surprisingly effective pseudo labels for the covariance. Our experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.