Enhancing Annotated Bibliography Generation with LLM Ensembles

TL;DR

This paper introduces a novel LLM ensemble approach for generating high-quality annotated bibliographies by combining multiple models with roles in generation, evaluation, and summarization, resulting in improved coherence and reduced redundancy.

Contribution

It presents a new ensemble methodology involving multiple LLM roles and validation strategies to enhance scholarly annotation tasks, which was not previously explored.

Findings

38% improvement in annotation quality

51% reduction in content redundancy

Enhanced coherence and relevance in outputs

Abstract

This work proposes a novel approach to enhancing annotated bibliography generation through Large Language Model (LLM) ensembles. In particular, multiple LLMs in different roles -- controllable text generation, evaluation, and summarization -- are introduced and validated using a systematic methodology to enhance model performance in scholarly tasks. Output diversity among the ensemble that generates text is obtained using different LLM parameters, followed by an LLM acting as a judge to assess relevance, accuracy, and coherence. Responses selected by several combining strategies are then merged and refined through summarization and redundancy removal techniques. The preliminary experimental validation demonstrates that the combined outputs from the LLM ensemble improve coherence and relevance compared to individual responses, leading to a 38% improvement in annotation quality and a 51% reduction in content redundancy, thus highlighting the potential for automating complex scholarly tasks while maintaining high-quality standards.