Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generation

TL;DR

The paper introduces a dual-phase framework for retrieval-augmented generation that assesses query clarity and model capability before deciding on retrieval, improving accuracy and efficiency in LLM applications.

Contribution

It proposes the extit{Think-then-Act} framework, a novel two-phase process that optimizes retrieval use by evaluating query and model confidence, reducing unnecessary retrievals.

Findings

Significant performance improvements on five datasets.

Enhanced accuracy and efficiency over baseline methods.

Effective in both English and non-English contexts.

Abstract

Despite their impressive capabilities, large language models (LLMs) often face challenges such as temporal misalignment and generating hallucinatory content. Enhancing LLMs with retrieval mechanisms to fetch relevant information from external sources offers a promising solution. Inspired by the proverb "Think twice before you act," we propose a dual-angle evaluated retrieval-augmented generation framework \textit{Think-then-Act}. Unlike previous approaches that indiscriminately rewrite queries or perform retrieval regardless of necessity, or generate temporary responses before deciding on additional retrieval, which increases model generation costs, our framework employs a two-phase process: (i) assessing the input query for clarity and completeness to determine if rewriting is necessary; and (ii) evaluating the model's capability to answer the query and deciding if additional retrieval is needed. Experimental results on five datasets show that the \textit{Think-then-Act} framework significantly improves performance. Our framework demonstrates notable improvements in accuracy and efficiency compared to existing baselines and performs well in both English and non-English contexts. Ablation studies validate the optimal model confidence threshold, highlighting the resource optimization benefits of our approach.