Scalable Bayesian Deep Learning with Kernel Seed Networks

TL;DR

This paper introduces Kernel Seed Networks, a scalable Bayesian deep learning method that reduces parameter count while maintaining uncertainty estimation quality, enabling safer AI applications in high-risk domains.

Contribution

The paper proposes Kernel Seed Networks, a novel approach that performs Bayesian deep learning without doubling parameters, outperforming traditional methods in efficiency and accuracy.

Findings

KSNs outperform conventional Bayesian methods

Parameter reduction up to 6.6 times

Effective uncertainty estimation in neural networks

Abstract

This paper addresses the scalability problem of Bayesian deep neural networks. The performance of deep neural networks is undermined by the fact that these algorithms have poorly calibrated measures of uncertainty. This restricts their application in high risk domains such as computer aided diagnosis and autonomous vehicle navigation. Bayesian Deep Learning (BDL) offers a promising method for representing uncertainty in neural network. However, BDL requires a separate set of parameters to store the mean and standard deviation of model weights to learn a distribution. This results in a prohibitive 2-fold increase in the number of model parameters. To address this problem we present a method for performing BDL, namely Kernel Seed Networks (KSN), which does not require a 2-fold increase in the number of parameters. KSNs use 1x1 Convolution operations to learn a compressed latent space representation of the parameter distribution. In this paper we show how this allows KSNs to outperform conventional BDL methods while reducing the number of required parameters by up to a factor of 6.6.