Leveraging Large Language Models for Comparative Literature Summarization with Reflective Incremental Mechanisms

TL;DR

This paper presents ChatCite, a novel LLM-based method for generating detailed comparative literature summaries through multi-step reasoning and reflective processes, outperforming existing models in automatic and human evaluations.

Contribution

Introduction of ChatCite, a multi-step, reflective LLM framework for comparative literature summarization, enhancing insightfulness and coherence over existing models.

Findings

ChatCite outperforms GPT-4, BART, T5, and CoT in automatic metrics.

Human evaluations favor ChatCite for coherence and insightfulness.

Significant improvement in literature review automation.

Abstract

In this paper, we introduce ChatCite, a novel method leveraging large language models (LLMs) for generating comparative literature summaries. The ability to summarize research papers with a focus on key comparisons between studies is an essential task in academic research. Existing summarization models, while effective at generating concise summaries, fail to provide deep comparative insights. ChatCite addresses this limitation by incorporating a multi-step reasoning mechanism that extracts critical elements from papers, incrementally builds a comparative summary, and refines the output through a reflective memory process. We evaluate ChatCite on a custom dataset, CompLit-LongContext, consisting of 1000 research papers with annotated comparative summaries. Experimental results show that ChatCite outperforms several baseline methods, including GPT-4, BART, T5, and CoT, across various automatic evaluation metrics such as ROUGE and the newly proposed G-Score. Human evaluation further confirms that ChatCite generates more coherent, insightful, and fluent summaries compared to these baseline models. Our method provides a significant advancement in automatic literature review generation, offering researchers a powerful tool for efficiently comparing and synthesizing scientific research.