In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

TL;DR

This paper introduces In-Context Transfer Learning (ICTL), a method that synthesizes target task demonstrations by transferring from similar source tasks, improving over scratch synthesis in large language models.

Contribution

ICTL proposes a transfer-based approach to synthesize task demonstrations, reducing reliance on costly labeled data and enhancing demonstration quality for in-context learning.

Findings

ICTL outperforms scratch synthesis by 2.0% on Super-NI.

Transfer learning improves demonstration quality for LLMs.

Method effectively reduces labeling costs.

Abstract

In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method.