SGIC: A Self-Guided Iterative Calibration Framework for RAG

TL;DR

The paper introduces SGIC, a novel self-guided iterative calibration framework that leverages uncertainty scores to improve the calibration and accuracy of large language models in retrieval-augmented generation tasks.

Contribution

It proposes a new calibration framework that uses uncertainty scores and iterative self-calibration to enhance LLM performance in RAG systems.

Findings

Significant performance improvements on multiple LLMs.

Effective calibration enhancement through iterative uncertainty scoring.

Applicable to both open-source and closed-source models.

Abstract

Recent research in retrieval-augmented generation (RAG) has concentrated on retrieving useful information from candidate documents. However, numerous methodologies frequently neglect the calibration capabilities of large language models (LLMs), which capitalize on their robust in-context reasoning prowess. This work illustrates that providing LLMs with specific cues substantially improves their calibration efficacy, especially in multi-round calibrations. We present a new SGIC: Self-Guided Iterative Calibration Framework that employs uncertainty scores as a tool. Initially, this framework calculates uncertainty scores to determine both the relevance of each document to the query and the confidence level in the responses produced by the LLMs. Subsequently, it reevaluates these scores iteratively, amalgamating them with prior responses to refine calibration. Furthermore, we introduce an innovative approach for constructing an iterative self-calibration training set, which optimizes LLMs to efficiently harness uncertainty scores for capturing critical information and enhancing response accuracy. Our proposed framework significantly improves performance on both closed-source and open-weight LLMs.