ISQA: Informative Factuality Feedback for Scientific Summarization

TL;DR

ISQA introduces an iterative feedback method that refines scientific summaries by reinforcing correct statements and correcting inaccuracies, significantly improving factuality in large language model outputs.

Contribution

It presents a novel feedback-based refinement approach for scientific summarization inspired by human learning theories, enhancing factual accuracy.

Findings

Significant factuality improvements across multiple datasets

Effective reinforcement of validated statements

Correction of inaccuracies in scientific summaries

Abstract

We propose Iterative Facuality Refining on Informative Scientific Question-Answering (ISQA) feedback\footnote{Code is available at \url{https://github.com/lizekai-richard/isqa}}, a method following human learning theories that employs model-generated feedback consisting of both positive and negative information. Through iterative refining of summaries, it probes for the underlying rationale of statements to enhance the factuality of scientific summarization. ISQA does this in a fine-grained manner by asking a summarization agent to reinforce validated statements in positive feedback and fix incorrect ones in negative feedback. Our findings demonstrate that the ISQA feedback mechanism significantly improves the factuality of various open-source LLMs on the summarization task, as evaluated across multiple scientific datasets.