TEGEE: Task dEfinition Guided Expert Ensembling for Generalizable and Few-shot Learning

TL;DR

TEGEE introduces a modular framework that explicitly extracts task definitions to improve few-shot and many-shot learning in LLMs, outperforming traditional in-context learning methods.

Contribution

The paper proposes TEGEE, a novel task definition guided ensembling method that emphasizes explicit task extraction and extends ICL to support unlimited demonstrations.

Findings

TEGEE performs comparably to larger models like LLaMA2-13B.

The modular design enhances continual learning and scalability.

Explicit task definition extraction is more crucial than task processing.

Abstract

Large Language Models (LLMs) exhibit the ability to perform in-context learning (ICL), where they acquire new tasks directly from examples provided in demonstrations. This process is thought to operate through an implicit task selection mechanism that involves extracting and processing task definitions from these demonstrations. However, critical questions remain: Which is more essential -- task extraction or definition? And how can these capabilities be further improved? To address these questions, we propose \textbf{TEGEE} (Task Definition Guided Expert Ensembling), a method that explicitly extracts task definitions and generates responses based on specific tasks. Our framework employs a dual 3B model approach, with each model assigned a distinct role: one focuses on task definition extraction, while the other handles learning from demonstrations. This modular approach supports the hypothesis that extracting task definitions is more vital than processing the task itself. Empirical evaluations show that TEGEE performs comparably to the larger LLaMA2-13B model. By leveraging a modular design, our approach extends traditional ICL from few-shot to many-shot learning, supporting an unlimited number of demonstrations and enhancing continual learning capabilities.