ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary

TL;DR

ChatCite is an LLM-based agent that mimics human workflow to generate comparative literature summaries, incorporating a reflective mechanism and an automatic evaluation metric, G-Score, to improve quality and usability.

Contribution

Introduces ChatCite, an innovative LLM agent with human workflow guidance and a new evaluation metric for more effective literature summarization.

Findings

ChatCite outperforms other models in multiple evaluation dimensions.

The generated summaries are suitable for drafting literature reviews.

The G-Score metric correlates well with human evaluations.

Abstract

The literature review is an indispensable step in the research process. It provides the benefit of comprehending the research problem and understanding the current research situation while conducting a comparative analysis of prior works. However, literature summary is challenging and time consuming. The previous LLM-based studies on literature review mainly focused on the complete process, including literature retrieval, screening, and summarization. However, for the summarization step, simple CoT method often lacks the ability to provide extensive comparative summary. In this work, we firstly focus on the independent literature summarization step and introduce ChatCite, an LLM agent with human workflow guidance for comparative literature summary. This agent, by mimicking the human workflow, first extracts key elements from relevant literature and then generates summaries using a Reflective Incremental Mechanism. In order to better evaluate the quality of the generated summaries, we devised a LLM-based automatic evaluation metric, G-Score, in refer to the human evaluation criteria. The ChatCite agent outperformed other models in various dimensions in the experiments. The literature summaries generated by ChatCite can also be directly used for drafting literature reviews.