Correcting Model Bias with Sparse Implicit Processes

TL;DR

This paper demonstrates that Sparse Implicit Processes (SIP) can effectively correct model bias in Bayesian machine learning, especially when the data-generating process differs from the model assumptions, leading to more accurate predictive distributions.

Contribution

The paper extends previous work on SIP, showing its ability to correct model bias in cases of strong model-data mismatch using synthetic datasets.

Findings

SIP provides better data reflection than initial model predictions.

SIP effectively corrects bias when model assumptions are strongly violated.

Synthetic experiments validate SIP's bias correction capabilities.

Abstract

Model selection in machine learning (ML) is a crucial part of the Bayesian learning procedure. Model choice may impose strong biases on the resulting predictions, which can hinder the performance of methods such as Bayesian neural networks and neural samplers. On the other hand, newly proposed approaches for Bayesian ML exploit features of approximate inference in function space with implicit stochastic processes (a generalization of Gaussian processes). The approach of Sparse Implicit Processes (SIP) is particularly successful in this regard, since it is fully trainable and achieves flexible predictions. Here, we expand on the original experiments to show that SIP is capable of correcting model bias when the data generating mechanism differs strongly from the one implied by the model. We use synthetic datasets to show that SIP is capable of providing predictive distributions that reflect the data better than the exact predictions of the initial, but wrongly assumed model.