EXnet: Efficient In-context Learning for Data-less Text classification

TL;DR

EXnet is a novel model that enables efficient in-context learning for text classification without example limitations, demonstrating strong cross-task generalization even at small sizes.

Contribution

We introduce EXnet, a model designed for unlimited in-context learning in text classification, enhancing generalization across tasks and domains.

Findings

Smallest model (15M parameters) generalizes well to unseen tasks.

In-context learning improves task accuracy and cross-task transfer.

EXnet outperforms traditional fine-tuning methods.

Abstract

Large pre-trained language models (PLMs) have made significant progress in encoding world knowledge and spawned a new set of learning paradigms including zero-shot, few-shot, and in-context learning. Many language tasks can be modeled as a set of prompts (for example, is this text about geography?) and language models can provide binary answers, i.e., Yes or No. There is evidence to suggest that the next-word prediction used by many PLMs does not align well with zero-shot paradigms. Therefore, PLMs are fine-tuned as a question-answering system. In-context learning extends zero-shot learning by incorporating prompts and examples, resulting in increased task accuracy. Our paper presents EXnet, a model specifically designed to perform in-context learning without any limitations on the number of examples. We argue that in-context learning is an effective method to increase task accuracy, and providing examples facilitates cross-task generalization, especially when it comes to text classification tasks. With extensive experiments, we show that even our smallest model (15M parameters) generalizes to several unseen classification tasks and domains.