QUILL: Quotation Generation Enhancement of Large Language Models

TL;DR

This paper introduces QUILL, a system that enhances large language models' ability to generate accurate and relevant quotations by establishing a new evaluation framework, creating a comprehensive bilingual quote database, and developing a reranking metric.

Contribution

The paper presents a novel evaluation system, a large bilingual quote knowledge base, and a quotation-specific reranking metric to improve LLMs' quotation generation performance.

Findings

Metrics strongly correlate with human preferences.

The knowledge base and reranking improve quote relevance.

Existing LLMs struggle with quote generation, but our methods help.

Abstract

While Large language models (LLMs) have become excellent writing assistants, they still struggle with quotation generation. This is because they either hallucinate when providing factual quotations or fail to provide quotes that exceed human expectations. To bridge the gap, we systematically study how to evaluate and improve LLMs' performance in quotation generation tasks. We first establish a holistic and automatic evaluation system for quotation generation task, which consists of five criteria each with corresponding automatic metric. To improve the LLMs' quotation generation abilities, we construct a bilingual knowledge base that is broad in scope and rich in dimensions, containing up to 32,022 quotes. Moreover, guided by our critiria, we further design a quotation-specific metric to rerank the retrieved quotations from the knowledge base. Extensive experiments show that our metrics strongly correlate with human preferences. Existing LLMs struggle to generate desired quotes, but our quotation knowledge base and reranking metric help narrow this gap. Our dataset and code are publicly available at https://github.com/GraceXiaoo/QUILL.