Theoretical Models of Learning to Learn

TL;DR

This paper introduces two theoretical models for learning to learn, demonstrating how machines can autonomously develop biases through exposure to related tasks within an environment.

Contribution

It presents the first formal PAC-type and hierarchical Bayes models for bias learning, advancing theoretical understanding of learning to learn.

Findings

PAC-type model based on empirical process theory

Hierarchical Bayes model for bias learning

Theoretical results demonstrating model capabilities

Abstract

A Machine can only learn if it is biased in some way. Typically the bias is supplied by hand, for example through the choice of an appropriate set of features. However, if the learning machine is embedded within an {\em environment} of related tasks, then it can {\em learn} its own bias by learning sufficiently many tasks from the environment. In this paper two models of bias learning (or equivalently, learning to learn) are introduced and the main theoretical results presented. The first model is a PAC-type model based on empirical process theory, while the second is a hierarchical Bayes model.