Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions

TL;DR

This paper introduces a demonstration notebook framework that automatically selects the most suitable in-context learning examples for each question, significantly improving performance across reasoning and summarization tasks.

Contribution

It proposes a novel prompt engineering workflow using demonstration notebooks to tailor in-context examples, outperforming existing methods and providing insights into demonstration effectiveness.

Findings

Outperforms all existing automatic demonstration selection methods

Achieves state-of-the-art results on reasoning benchmarks

Effective in text summarization and prompt compression tasks

Abstract

Large language models (LLMs) benefit greatly from prompt engineering, with in-context learning standing as a pivital technique. While former approaches have provided various ways to construct the demonstrations used for in-context learning, they often ignore the inherent heterogeneity within datasets, applying the same demonstrations to all reasoning questions. We observed that the effectiveness of demonstrations varies depending on the specific question. This motivates our exploration of using prompt engineering to select appropriate demonstrations. To address the challenge of automatically creating and choosing demonstrations tailored to each question, we propose a novel prompt engineering workflow built around a novel object called the "demonstration notebook." This notebook helps identify the most suitable in-context learning example for a question by gathering and reusing information from the LLM's past interactions. Our experiments show that this approach outperforms all existing methods for automatic demonstration construction and selection (as far as we know), achieving state-of-the-art results on serveral reasoning benchmarks. The method's versatility is further demonstrated by its success in text summarization and prompt compression tasks. Additionally, we contribute a rigorous analysis method to reveal the "demonstrative regime" of a demonstration, providing valuable insights into how demonstrations relate to different question types within a dataset.