Curriculum Learning of Multiple Tasks

TL;DR

This paper introduces a curriculum learning approach for multi-task learning that sequences tasks based on a generalization bound criterion, improving overall performance by discovering optimal task orders.

Contribution

It proposes a novel method for sequential multi-task learning that automatically identifies the best task order using a generalization bound, outperforming joint learning.

Findings

Sequential learning can outperform joint multi-task learning.

Task order significantly impacts performance.

The method automatically finds favorable task sequences.

Abstract

Sharing information between multiple tasks enables algorithms to achieve good generalization performance even from small amounts of training data. However, in a realistic scenario of multi-task learning not all tasks are equally related to each other, hence it could be advantageous to transfer information only between the most related tasks. In this work we propose an approach that processes multiple tasks in a sequence with sharing between subsequent tasks instead of solving all tasks jointly. Subsequently, we address the question of curriculum learning of tasks, i.e. finding the best order of tasks to be learned. Our approach is based on a generalization bound criterion for choosing the task order that optimizes the average expected classification performance over all tasks. Our experimental results show that learning multiple related tasks sequentially can be more effective than learning them jointly, the order in which tasks are being solved affects the overall performance, and that our model is able to automatically discover the favourable order of tasks.