Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answering

TL;DR

This paper introduces AttenHScore, a real-time hallucination detection metric for small language models in question answering, enabling adaptive invocation of large models to improve accuracy and reduce costs.

Contribution

We propose AttenHScore for dynamic hallucination detection and uncertainty-aware knowledge reorganization, improving real-time invocation of large LMs without extra training.

Findings

AttenHScore outperforms baselines in hallucination detection across multiple datasets.

Our methods reduce reliance on additional model training.

Strategies effectively handle complex queries with improved accuracy.

Abstract

The collaborative paradigm of large and small language models (LMs) effectively balances performance and cost, yet its pivotal challenge lies in precisely pinpointing the moment of invocation when hallucinations arise in small LMs. Previous optimization efforts primarily focused on post-processing techniques, which were separate from the reasoning process of LMs, resulting in high computational costs and limited effectiveness. In this paper, we propose a practical invocation evaluation metric called AttenHScore, which calculates the accumulation and propagation of hallucinations during the generation process of small LMs, continuously amplifying potential reasoning errors. By dynamically adjusting the detection threshold, we achieve more accurate real-time invocation of large LMs. Additionally, considering the limited reasoning capacity of small LMs, we leverage uncertainty-aware knowledge reorganization to assist them better capture critical information from different text chunks. Extensive experiments reveal that our AttenHScore outperforms most baselines in enhancing real-time hallucination detection capabilities across multiple QA datasets, especially when addressing complex queries. Moreover, our strategies eliminate the need for additional model training and display flexibility in adapting to various transformer-based LMs.