Are Human-generated Demonstrations Necessary for In-context Learning?

TL;DR

This paper introduces SEC, a prompting strategy where large language models generate their own demonstrations, eliminating the need for human-crafted examples and achieving comparable or better performance in various reasoning tasks.

Contribution

SEC is a novel framework enabling LLMs to self-generate demonstrations, reducing reliance on manual examples and simplifying the in-context learning process.

Findings

SEC outperforms zero-shot learning across tasks.

SEC matches the performance of traditional ICL with human demonstrations.

SEC simplifies the process of in-context learning for LLMs.

Abstract

Despite the promising few-shot ability of large language models (LLMs), the standard paradigm of In-context Learning (ICL) suffers the disadvantages of susceptibility to selected demonstrations and the intricacy to generate these demonstrations. In this paper, we raise the fundamental question that whether human-generated demonstrations are necessary for ICL. To answer this question, we propose self-contemplation prompting strategy (SEC), a paradigm free from human-crafted demonstrations. The key point of SEC is that, instead of using hand-crafted examples as demonstrations in ICL, SEC asks LLMs to first create demonstrations on their own, based on which the final output is generated. SEC is a flexible framework and can be adapted to both the vanilla ICL and the chain-of-thought (CoT), but with greater ease: as the manual-generation process of both examples and rationale can be saved. Extensive experiments in arithmetic reasoning, commonsense reasoning, multi-task language understanding, and code generation benchmarks, show that SEC, which does not require hand-crafted demonstrations, significantly outperforms the zero-shot learning strategy, and achieves comparable results to ICL with hand-crafted demonstrations. This demonstrates that, for many tasks, contemporary LLMs possess a sufficient level of competence to exclusively depend on their own capacity for decision making, removing the need for external training data. Code is available at https://github.com/ruili33/SEC.