ScoreRAG: A Retrieval-Augmented Generation Framework with Consistency-Relevance Scoring and Structured Summarization for News Generation

TL;DR

ScoreRAG is a novel framework that combines retrieval, relevance scoring, and structured summarization to improve the accuracy, coherence, and professionalism of automated news generation, addressing hallucinations and factual inconsistencies.

Contribution

It introduces a multi-stage retrieval and scoring framework that enhances news generation quality by filtering and guiding large language models with relevance and consistency assessments.

Findings

Improved factual accuracy in generated news articles

Enhanced coherence and professionalism in outputs

Effective filtering of low-quality information

Abstract

This research introduces ScoreRAG, an approach to enhance the quality of automated news generation. Despite advancements in Natural Language Processing and large language models, current news generation methods often struggle with hallucinations, factual inconsistencies, and lack of domain-specific expertise when producing news articles. ScoreRAG addresses these challenges through a multi-stage framework combining retrieval-augmented generation, consistency relevance evaluation, and structured summarization. The system first retrieves relevant news documents from a vector database, maps them to complete news items, and assigns consistency relevance scores based on large language model evaluations. These documents are then reranked according to relevance, with low-quality items filtered out. The framework proceeds to generate graded summaries based on relevance scores, which guide the large language model in producing complete news articles following professional journalistic standards. Through this methodical approach, ScoreRAG aims to significantly improve the accuracy, coherence, informativeness, and professionalism of generated news articles while maintaining stability and consistency throughout the generation process. The code and demo are available at: https://github.com/peiyun2260/ScoreRAG.