Teaching language models to support answers with verified quotes

TL;DR

This paper introduces GopherCite, a large language model trained with reinforcement learning to generate factually supported answers with citations, improving trustworthiness and allowing abstention on uncertain questions.

Contribution

It presents a novel training method combining reinforcement learning and evidence citation for large language models to enhance answer reliability.

Findings

GopherCite achieves 80% high-quality answers on NaturalQuestions.

It attains 67% quality on ELI5, improving to 90% and 80% with abstention.

Citation alone does not guarantee truthfulness, highlighting safety challenges.

Abstract

Recent large language models often answer factual questions correctly. But users can't trust any given claim a model makes without fact-checking, because language models can hallucinate convincing nonsense. In this work we use reinforcement learning from human preferences (RLHP) to train "open-book" QA models that generate answers whilst also citing specific evidence for their claims, which aids in the appraisal of correctness. Supporting evidence is drawn from multiple documents found via a search engine, or from a single user-provided document. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure. We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets. The model's response is found to be high-quality 80\% of the time on this Natural Questions subset, and 67\% of the time on the ELI5 subset. Abstaining from the third of questions for which it is most unsure improves performance to 90\% and 80\% respectively, approaching human baselines. However, analysis on the adversarial TruthfulQA dataset shows why citation is only one part of an overall strategy for safety and trustworthiness: not all claims supported by evidence are true.