Loss-Calibrated Approximate Inference in Bayesian Neural Networks

TL;DR

This paper introduces a loss-calibrated Bayesian neural network approach that incorporates task-specific utility functions into the inference process, improving prediction utility especially in applications with asymmetric utilities.

Contribution

It proposes a novel loss-calibrated evidence lower bound for Bayesian neural networks, integrating utility functions into the inference to optimize task-specific performance.

Findings

Higher utility in task-specific predictions compared to traditional methods

Effective in medical diagnosis and autonomous driving scenarios

Identifies failure modes of weighted cross entropy approach

Abstract

Current approaches in approximate inference for Bayesian neural networks minimise the Kullback-Leibler divergence to approximate the true posterior over the weights. However, this approximation is without knowledge of the final application, and therefore cannot guarantee optimal predictions for a given task. To make more suitable task-specific approximations, we introduce a new loss-calibrated evidence lower bound for Bayesian neural networks in the context of supervised learning, informed by Bayesian decision theory. By introducing a lower bound that depends on a utility function, we ensure that our approximation achieves higher utility than traditional methods for applications that have asymmetric utility functions. Furthermore, in using dropout inference, we highlight that our new objective is identical to that of standard dropout neural networks, with an additional utility-dependent penalty term. We demonstrate our new loss-calibrated model with an illustrative medical example and a restricted model capacity experiment, and highlight failure modes of the comparable weighted cross entropy approach. Lastly, we demonstrate the scalability of our method to real world applications with per-pixel semantic segmentation on an autonomous driving data set.