Formal Bayesian Transfer Learning via the Total Risk Prior

TL;DR

This paper introduces a Bayesian transfer learning method that constructs a joint prior from multiple source datasets, enabling full uncertainty quantification and improved predictions, especially with limited source data.

Contribution

It proposes a novel Bayesian framework using risk minimizers as priors, allowing model averaging and better handling of limited, misaligned source datasets.

Findings

Outperforms frequentist methods in genetics prediction tasks.

Enables Bayesian uncertainty quantification and model averaging.

Links minimax-frequentist techniques to MAP estimation in the proposed model.

Abstract

In analyses with severe data-limitations, augmenting the target dataset with information from ancillary datasets in the application domain, called source datasets, can lead to significantly improved statistical procedures. However, existing methods for this transfer learning struggle to deal with situations where the source datasets are also limited and not guaranteed to be well-aligned with the target dataset. A typical strategy is to use the empirical loss minimizer on the source data as a prior mean for the target parameters, which places the estimation of source parameters outside of the Bayesian formalism. Our key conceptual contribution is to use a risk minimizer conditional on source parameters instead. This allows us to construct a single joint prior distribution for all parameters from the source datasets as well as the target dataset. As a consequence, we benefit from full Bayesian uncertainty quantification and can perform model averaging via Gibbs sampling over indicator variables governing the inclusion of each source dataset. We show how a particular instantiation of our prior leads to a Bayesian Lasso in a transformed coordinate system and discuss computational techniques to scale our approach to moderately sized datasets. We also demonstrate that recently proposed minimax-frequentist transfer learning techniques may be viewed as an approximate Maximum a Posteriori approach to our model. Finally, we demonstrate superior predictive performance relative to the frequentist baseline on a genetics application, especially when the source data are limited.