CrossICL: Cross-Task In-Context Learning via Unsupervised Demonstration Transfer

TL;DR

CrossICL introduces a method to leverage existing source task demonstrations for new target tasks in in-context learning, reducing manual effort and improving performance across diverse NLP tasks.

Contribution

The paper proposes a novel CrossICL paradigm with a two-stage alignment strategy to effectively transfer demonstrations across tasks without additional manual input.

Findings

CrossICL significantly improves performance on 875 NLP tasks.

Effective demonstration transfer depends on task similarity and gap characteristics.

The approach reduces the need for manual demonstration collection.

Abstract

In-Context Learning (ICL) enhances the performance of large language models (LLMs) with demonstrations. However, obtaining these demonstrations primarily relies on manual effort. In most real-world scenarios, users are often unwilling or unable to provide such demonstrations. Inspired by the human analogy, we explore a new ICL paradigm CrossICL to study how to utilize existing source task demonstrations in the ICL for target tasks, thereby obtaining reliable guidance without any additional manual effort. To explore this, we first design a two-stage alignment strategy to mitigate the interference caused by gaps across tasks, as the foundation for our experimental exploration. Based on it, we conduct comprehensive exploration of CrossICL, with 875 NLP tasks from the Super-NI benchmark and six types of LLMs, including GPT-4o. Experimental results demonstrate the effectiveness of CrossICL and provide valuable insights on questions like the criteria for selecting cross-task demonstrations, as well as the types of task-gap-induced interference in CrossICL.