Xinyu: An Efficient LLM-based System for Commentary Generation

TL;DR

Xinyu is an efficient LLM-based system that streamlines Chinese commentary generation by decomposing the task, employing targeted strategies, and utilizing retrieval-augmented evidence, significantly reducing creation time without sacrificing quality.

Contribution

The paper introduces a novel, step-by-step LLM-based framework with fine-tuning, argument ranking, and retrieval augmentation to improve commentary generation efficiency and quality.

Findings

Commentary creation time reduced from 4 hours to 20 minutes.

System achieves well-structured, logical, and evidence-backed commentaries.

Evaluation metric considers multiple perspectives for fair assessment.

Abstract

Commentary provides readers with a deep understanding of events by presenting diverse arguments and evidence. However, creating commentary is a time-consuming task, even for skilled commentators. Large language models (LLMs) have simplified the process of natural language generation, but their direct application in commentary creation still faces challenges due to unique task requirements. These requirements can be categorized into two levels: 1) fundamental requirements, which include creating well-structured and logically consistent narratives, and 2) advanced requirements, which involve generating quality arguments and providing convincing evidence. In this paper, we introduce Xinyu, an efficient LLM-based system designed to assist commentators in generating Chinese commentaries. To meet the fundamental requirements, we deconstruct the generation process into sequential steps, proposing targeted strategies and supervised fine-tuning (SFT) for each step. To address the advanced requirements, we present an argument ranking model for arguments and establish a comprehensive evidence database that includes up-to-date events and classic books, thereby strengthening the substantiation of the evidence with retrieval augmented generation (RAG) technology. To evaluate the generated commentaries more fairly, corresponding to the two-level requirements, we introduce a comprehensive evaluation metric that considers five distinct perspectives in commentary generation. Our experiments confirm the effectiveness of our proposed system. We also observe a significant increase in the efficiency of commentators in real-world scenarios, with the average time spent on creating a commentary dropping from 4 hours to 20 minutes. Importantly, such an increase in efficiency does not compromise the quality of the commentaries.