A Bayesian/Information Theoretic Model of Bias Learning

TL;DR

This paper explores how Bayesian methods can be used to learn appropriate biases for related tasks in machine learning, emphasizing the role of objective priors and the benefits of multi-task learning in reducing information requirements.

Contribution

It introduces a Bayesian framework for learning task biases using objective priors and provides bounds on information needed for multi-task learning when the true prior is unknown.

Findings

Sampling multiple tasks reduces the information needed to learn the true prior.

Small-dimensional true priors are easier to learn with limited knowledge.

Bayesian inference can identify the true prior within a set of possible priors.

Abstract

In this paper the problem of learning appropriate bias for an environment of related tasks is examined from a Bayesian perspective. The environment of related tasks is shown to be naturally modelled by the concept of an {\em objective} prior distribution. Sampling from the objective prior corresponds to sampling different learning tasks from the environment. It is argued that for many common machine learning problems, although we don't know the true (objective) prior for the problem, we do have some idea of a set of possible priors to which the true prior belongs. It is shown that under these circumstances a learner can use Bayesian inference to learn the true prior by sampling from the objective prior. Bounds are given on the amount of information required to learn a task when it is simultaneously learnt with several other tasks. The bounds show that if the learner has little knowledge of the true prior, and the dimensionality of the true prior is small, then sampling multiple tasks is highly advantageous.