Can we only use guideline instead of shot in prompt?

TL;DR

This paper introduces the FGT framework that automatically learns and utilizes task-specific guidelines in prompts, replacing traditional shot-based methods to improve reasoning performance across multiple tasks.

Contribution

The paper proposes a novel FGT framework that automatically extracts and aggregates guidelines from feedback, enabling guideline-based prompting as an effective alternative to shot-based methods.

Findings

FGT achieves superior performance on multiple tasks.

Guideline-based prompting reduces the complexity of prompt design.

The framework effectively learns and consolidates task-specific guidelines.

Abstract

Currently, prompting techniques can be mainly divided into two categories:1)shot method implicitly inspires the model to answer the question by mimicing the steps in the given example, e.g., the few-shot CoT. 2) Guideline method explicitly instructs the model to reason by following guidelines, which contains succinct and concise task-specific knowledge. Shot method is prone to difficulties in terms of selection of shots type, the number of shots, and the design of the reasoning steps, so a question arises: can we only use guideline instead of shot in the prompt? To this end, we propose the FGT framework to automatically learn task-specific guidelines from dataset consisting of Feedback, Guideline, and Tree-gather agents. First, the feedback agent is designed to evaluate the outcomes, both right and wrong, of each Q&A to gather insights guiding more effective optimization strategies. Next, the guideline agent is tasked with deriving guidelines from each piece of feedback and storing them in local memory. Lastly, the tree-gather agent aggregates all guidelines hierarchically through a tree structure, ultimately obtaining all unduplicated guidelines from a global perspective. In addition, we induce the model to generate intermediate processes to ensure the reasoning consistent with the guidelines. Experimental results demonstrate that our approach achieves superior performance across multiple tasks, thereby highlighting the effectiveness of using the guidelines in prompt.