The Epistemic Uncertainty Gradient in Spaces of Random Projections

TL;DR

This paper introduces a new way to use uncertainty estimates in machine learning to better understand and work with data distributions.

Contribution

The novel contribution is reinterpreting epistemic uncertainty as a model of the data distribution using random projections.

Findings

The approach enables efficient and parameter-free representation of arbitrary data distributions.

Minimizing uncertainty via gradient descent helps find data points close to an initial input within the distribution.

Applications include local Gaussian approximations, regression, and data unlearning.

Abstract

This work presents a novel approach to handling epistemic uncertainty estimates with motivation from Bayesian linear regression. We propose treating the model-dependent variance in the predictive distribution—commonly associated with epistemic uncertainty—as a model for the underlying data distribution. Using high-dimensional random feature transformations, this approach allows for a computationally efficient, parameter-free representation of arbitrary data distributions. This allows assessing whether a query point lies within the distribution, which can also provide insights into outlier detection and generalization tasks. Furthermore, given an initial input, minimizing the uncertainty using gradient descent offers a new method of querying data points that are close to the initial input and belong to the distribution resembling the training data, much like auto-completion in associative networks. We extend the proposed method to applications such as local Gaussian approximations, input–output regression, and even a mechanism for unlearning of data. This reinterpretation of uncertainty, alongside the geometric insights it provides, offers an innovative and novel framework for addressing classical machine learning challenges.

Figures