Label Budget Allocation in Multi-Task Learning

TL;DR

This paper introduces the label budget allocation problem in multi-task learning, proposing a task-adaptive algorithm that optimizes label distribution among tasks to improve overall performance, validated on PASCAL VOC and Taskonomy datasets.

Contribution

It formally defines the label budget allocation problem in multi-task learning and presents a novel adaptive algorithm to optimize label distribution for better performance.

Findings

Different budget strategies significantly impact performance.

The proposed algorithm outperforms heuristic strategies.

Effective budget allocation improves multi-task learning results.

Abstract

The cost of labeling data often limits the performance of machine learning systems. In multi-task learning, related tasks provide information to each other and improve overall performance, but the label cost can vary among tasks. How should the label budget (i.e. the amount of money spent on labeling) be allocated among different tasks to achieve optimal multi-task performance? We are the first to propose and formally define the label budget allocation problem in multi-task learning and to empirically show that different budget allocation strategies make a big difference to its performance. We propose a Task-Adaptive Budget Allocation algorithm to robustly generate the optimal budget allocation adaptive to different multi-task learning settings. Specifically, we estimate and then maximize the extent of new information obtained from the allocated budget as a proxy for multi-task learning performance. Experiments on PASCAL VOC and Taskonomy demonstrate the efficacy of our approach over other widely used heuristic labeling strategies.