Sparse Function-space Representation of Neural Networks

TL;DR

This paper introduces a dual parameterization method that converts neural networks into a sparse function-space representation, improving uncertainty estimation and data incorporation without retraining.

Contribution

It presents a novel dual parameterization enabling sparse, data-efficient function-space representations of neural networks for uncertainty quantification.

Findings

Effective uncertainty quantification demonstrated on UCI benchmarks.

Enables incorporation of new data without retraining.

Maintains predictive performance with sparse representations.

Abstract

Deep neural networks (NNs) are known to lack uncertainty estimates and struggle to incorporate new data. We present a method that mitigates these issues by converting NNs from weight space to function space, via a dual parameterization. Importantly, the dual parameterization enables us to formulate a sparse representation that captures information from the entire data set. This offers a compact and principled way of capturing uncertainty and enables us to incorporate new data without retraining whilst retaining predictive performance. We provide proof-of-concept demonstrations with the proposed approach for quantifying uncertainty in supervised learning on UCI benchmark tasks.