Deep Reference Priors: What is the best way to pretrain a model?

TL;DR

This paper introduces the concept of deep reference priors, leveraging Bayesian theory to optimize pretraining and semi-supervised learning in deep networks, especially with limited data, through the use of objective, data-dependent priors.

Contribution

It formalizes reference priors for deep networks, demonstrating their application in semi-supervised learning and transfer learning for the first time.

Findings

Effective semi-supervised learning with few samples per class

Improved transfer learning by using source task data for prior computation

Validated on image classification datasets

Abstract

What is the best way to exploit extra data -- be it unlabeled data from the same task, or labeled data from a related task -- to learn a given task? This paper formalizes the question using the theory of reference priors. Reference priors are objective, uninformative Bayesian priors that maximize the mutual information between the task and the weights of the model. Such priors enable the task to maximally affect the Bayesian posterior, e.g., reference priors depend upon the number of samples available for learning the task and for very small sample sizes, the prior puts more probability mass on low-complexity models in the hypothesis space. This paper presents the first demonstration of reference priors for medium-scale deep networks and image-based data. We develop generalizations of reference priors and demonstrate applications to two problems. First, by using unlabeled data to compute the reference prior, we develop new Bayesian semi-supervised learning methods that remain effective even with very few samples per class. Second, by using labeled data from the source task to compute the reference prior, we develop a new pretraining method for transfer learning that allows data from the target task to maximally affect the Bayesian posterior. Empirical validation of these methods is conducted on image classification datasets. Code is available at https://github.com/grasp-lyrl/deep_reference_priors.