ICL Markup: Structuring In-Context Learning using Soft-Token Tags

TL;DR

This paper introduces ICL Markup, a method using soft-token tags to structure in-context learning prompts, reducing arbitrary choices and improving performance on various tasks through a meta-learning approach.

Contribution

It proposes a novel soft-token tagging method for ICL prompt design, learned via parameter-efficient fine-tuning, enhancing adaptability without extra training on new tasks.

Findings

Improved LLM performance on intent detection and text classification tasks.

Reduces arbitrary prompt design decisions in ICL.

Promising results in enterprise applications.

Abstract

Large pretrained language models (LLMs) can be rapidly adapted to a wide variety of tasks via a text-to-text approach, where the instruction and input are fed to the model in natural language. Combined with in-context learning (ICL), this paradigm is impressively flexible and powerful. However, it also burdens users with an overwhelming number of choices, many of them arbitrary. Inspired by markup languages like HTML, we contribute a method of using soft-token tags to compose prompt templates. This approach reduces arbitrary decisions and streamlines the application of ICL. Our method is a form of meta-learning for ICL; it learns these tags in advance during a parameter-efficient fine-tuning ``warm-up'' process. The tags can subsequently be used in templates for ICL on new, unseen tasks without any additional fine-tuning. Our experiments with this approach yield promising initial results, improving LLM performance on important enterprise applications such as few-shot and open-world intent detection, as well as text classification in news and legal domains.