How to distribute data across tasks for meta-learning?

TL;DR

This paper investigates optimal data distribution strategies across tasks in meta-learning, revealing how task difficulty and homogeneity influence label allocation to improve learning efficiency and transferability.

Contribution

It provides a theoretical and empirical analysis of label allocation in meta-learning, offering guidelines for data distribution based on task similarity and difficulty.

Findings

Uniform allocation is optimal for homogeneous tasks.

Trade-off exists between number of tasks and data per task at fixed budget.

Harder tasks benefit from more data when trained separately.

Abstract

Meta-learning models transfer the knowledge acquired from previous tasks to quickly learn new ones. They are trained on benchmarks with a fixed number of data points per task. This number is usually arbitrary and it is unknown how it affects performance at testing. Since labelling of data is expensive, finding the optimal allocation of labels across training tasks may reduce costs. Given a fixed budget of labels, should we use a small number of highly labelled tasks, or many tasks with few labels each? Should we allocate more labels to some tasks and less to others? We show that: 1) If tasks are homogeneous, there is a uniform optimal allocation, whereby all tasks get the same amount of data; 2) At fixed budget, there is a trade-off between number of tasks and number of data points per task, with a unique solution for the optimum; 3) When trained separately, harder task should get more data, at the cost of a smaller number of tasks; 4) When training on a mixture of easy and hard tasks, more data should be allocated to easy tasks. Interestingly, Neuroscience experiments have shown that human visual skills also transfer better from easy tasks. We prove these results mathematically on mixed linear regression, and we show empirically that the same results hold for few-shot image classification on CIFAR-FS and mini-ImageNet. Our results provide guidance for allocating labels across tasks when collecting data for meta-learning.