Bayesian Linear Regression on Deep Representations

TL;DR

This paper introduces a heteroscedastic Bayesian linear regression method on deep representations, improving uncertainty estimation for regression tasks and demonstrating competitive performance in benchmarks and reinforcement learning.

Contribution

It presents a novel heteroscedastic extension to Bayesian linear regression on deep features, enhancing uncertainty modeling capabilities.

Findings

The method effectively models heteroscedastic noise.

It performs competitively with ensembling methods.

Ensembles of the proposed method outperform other approaches.

Abstract

A simple approach to obtaining uncertainty-aware neural networks for regression is to do Bayesian linear regression (BLR) on the representation from the last hidden layer. Recent work [Riquelme et al., 2018, Azizzadenesheli et al., 2018] indicates that the method is promising, though it has been limited to homoscedastic noise. In this paper, we propose a novel variation that enables the method to flexibly model heteroscedastic noise. The method is benchmarked against two prominent alternative methods on a set of standard datasets, and finally evaluated as an uncertainty-aware model in model-based reinforcement learning. Our experiments indicate that the method is competitive with standard ensembling, and ensembles of BLR outperforms the methods we compared to.