Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP

TL;DR

This paper introduces diverse self-supervised task distributions for meta-learning in NLP, significantly improving few-shot learning performance by leveraging unlabeled text to create varied training tasks.

Contribution

It proposes a method to generate multiple self-supervised task distributions considering diversity, difficulty, and domain, enhancing meta-learning effectiveness in NLP.

Findings

Up to +4.2% absolute accuracy in few-shot learning

Achieves comparable results to supervised methods on FewRel 2.0

Demonstrates the importance of task diversity and curriculum in meta-learning

Abstract

Meta-learning considers the problem of learning an efficient learning process that can leverage its past experience to accurately solve new tasks. However, the efficacy of meta-learning crucially depends on the distribution of tasks available for training, and this is often assumed to be known a priori or constructed from limited supervised datasets. In this work, we aim to provide task distributions for meta-learning by considering self-supervised tasks automatically proposed from unlabeled text, to enable large-scale meta-learning in NLP. We design multiple distributions of self-supervised tasks by considering important aspects of task diversity, difficulty, type, domain, and curriculum, and investigate how they affect meta-learning performance. Our analysis shows that all these factors meaningfully alter the task distribution, some inducing significant improvements in downstream few-shot accuracy of the meta-learned models. Empirically, results on 20 downstream tasks show significant improvements in few-shot learning -- adding up to +4.2% absolute accuracy (on average) to the previous unsupervised meta-learning method, and perform comparably to supervised methods on the FewRel 2.0 benchmark.