MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces

TL;DR

MetaXT introduces a novel bi-level optimization framework that enables effective transfer learning across NLP tasks with different label spaces, significantly improving performance in few-shot scenarios.

Contribution

The paper proposes MetaXT, a method that transfers knowledge between tasks with disparate label spaces using a label transfer network and bi-level optimization.

Findings

MetaXT outperforms baseline methods in cross-task transfer settings.

Effective in low-resource, few-shot learning scenarios.

Demonstrates robustness across four NLP tasks with label space disparities.

Abstract

Albeit the universal representational power of pre-trained language models, adapting them onto a specific NLP task still requires a considerably large amount of labeled data. Effective task fine-tuning meets challenges when only a few labeled examples are present for the task. In this paper, we aim to the address of the problem of few shot task learning by exploiting and transferring from a different task which admits a related but disparate label space. Specifically, we devise a label transfer network (LTN) to transform the labels from source task to the target task of interest for training. Both the LTN and the model for task prediction are learned via a bi-level optimization framework, which we term as MetaXT. MetaXT offers a principled solution to best adapt a pre-trained language model to the target task by transferring knowledge from the source task. Empirical evaluations on cross-task transfer settings for four NLP tasks, from two different types of label space disparities, demonstrate the effectiveness of MetaXT, especially when the labeled data in the target task is limited.