Learning Model Bias

TL;DR

This paper explores how learning multiple related tasks can help automatically acquire domain-specific biases, providing theoretical bounds and experimental evidence for improved generalization with multi-task learning.

Contribution

It introduces a theorem bounding the number of tasks needed to learn domain bias and shows how multi-task learning reduces sample complexity per task.

Findings

Number of tasks needed scales with domain complexity

Multi-task learning reduces examples needed per task

Experimental results support theoretical bounds

Abstract

In this paper the problem of {\em learning} appropriate domain-specific bias is addressed. It is shown that this can be achieved by learning many related tasks from the same domain, and a theorem is given bounding the number tasks that must be learnt. A corollary of the theorem is that if the tasks are known to possess a common {\em internal representation} or {\em preprocessing} then the number of examples required per task for good generalisation when learning $n$ tasks simultaneously scales like $O(a + \frac{b}{n})$, where $O(a)$ is a bound on the minimum number of examples required to learn a single task, and $O(a + b)$ is a bound on the number of examples required to learn each task independently. An experiment providing strong qualitative support for the theoretical results is reported.