Bayesian inference for the learning rate in Generalised Bayesian inference

TL;DR

This paper introduces a Bayesian approach to estimate hyperparameters in Generalised Bayesian Inference, enabling joint uncertainty quantification and improved hyperparameter selection.

Contribution

It defines two new hyperparameter posteriors based on utility and pseudo-true parameters, supporting joint estimation and uncertainty quantification.

Findings

GBI-posteriors outperform standard Bayesian inference on simulated data.

The method effectively selects near-optimal hyperparameters in text analysis.

Supports combining multiple datasets with uncertainty quantification.

Abstract

In Generalised Bayesian Inference (GBI), the learning rate and hyperparameters of the loss must be estimated. These inference-hyperparameters can't be estimated jointly with the other parameters, from the data, by giving them a prior. However, in some settings there exist unknown ``true'' hyperparameter-values about which it is meaningful to have prior belief. It is then possible to use Bayesian inference with held-out data to get hyperparameter-posteriors. We define two hyperparameter posteriors, one based on an ELPPD-utility and one aiming to cover the pseudo-true parameter. The new framework supports estimation and uncertainty quantification for multiple hyperparameters jointly. Experiments show that the resulting GBI-posteriors out-perform Bayesian inference on simulated test data and select optimal or near optimal hyperparameter values in a large real problem of text analysis. Generalised Bayesian inference is particularly useful for combining multiple data sets and most of our examples belong to that setting. We also give asymptotic results for some of the special ``multi-modular'' Generalised Bayes posteriors which we use in our examples.