LLM-Ref: Enhancing Reference Handling in Technical Writing with Large Language Models

TL;DR

LLM-Ref is a novel writing assistant that improves reference handling in technical writing by directly retrieving and generating content from source texts, significantly enhancing accuracy and relevance.

Contribution

We introduce LLM-Ref, a tool that enhances reference synthesis in technical writing by directly extracting references from text and managing lengthy contexts effectively.

Findings

Achieved 3.25x to 6.26x increase in Ragas score over baseline systems.

Facilitates direct reference extraction from generated outputs.

Effectively manages lengthy contexts within LLM constraints.

Abstract

Large Language Models (LLMs) excel in data synthesis but can be inaccurate in domain-specific tasks, which retrieval-augmented generation (RAG) systems address by leveraging user-provided data. However, RAGs require optimization in both retrieval and generation stages, which can affect output quality. In this paper, we present LLM-Ref, a writing assistant tool that aids researchers in writing articles from multiple source documents with enhanced reference synthesis and handling capabilities. Unlike traditional RAG systems that use chunking and indexing, our tool retrieves and generates content directly from text paragraphs. This method facilitates direct reference extraction from the generated outputs, a feature unique to our tool. Additionally, our tool employs iterative response generation, effectively managing lengthy contexts within the language model's constraints. Compared to baseline RAG-based systems, our approach achieves a $3.25\times$ to $6.26\times$ increase in Ragas score, a comprehensive metric that provides a holistic view of a RAG system's ability to produce accurate, relevant, and contextually appropriate responses. This improvement shows our method enhances the accuracy and contextual relevance of writing assistance tools.