LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations

TL;DR

LoVeC introduces a reinforcement learning approach to train large language models to append real-time, verbalized confidence scores during long-form generation, improving factuality detection efficiently.

Contribution

The paper presents LoVeC, a novel RL-based method enabling LLMs to generate verbalized confidence scores on-the-fly, enhancing factuality estimation in long-form content.

Findings

LoVeC achieves better calibration than self-consistency methods.

The method is 20 times faster than traditional approaches.

LoVeC generalizes robustly across different domains.

Abstract

Hallucination remains a major challenge for the safe and trustworthy deployment of large language models (LLMs) in factual content generation. Prior work has explored confidence estimation as an effective approach to hallucination detection, but often relies on post-hoc self-consistency methods that require computationally expensive sampling. Verbalized confidence offers a more efficient alternative, but existing approaches are largely limited to short-form question answering (QA) tasks and do not generalize well to open-ended generation. In this paper, we propose LoVeC (Long-form Verbalized Confidence), a novel reinforcement learning based method that trains LLMs to append an on-the-fly numerical confidence score to each generated statement during long-form generation. The confidence score serves as a direct and interpretable signal of the factuality of generation. We introduce two evaluation settings, free-form tagging and iterative tagging, to assess different verbalized confidence estimation methods. Experiments on three long-form QA datasets show that our RL-trained models achieve better calibration and generalize robustly across domains. Also, our method is highly efficient, being 20 times faster than traditional self-consistency methods while achieving better calibration.