Multi-LLM Collaborative Caption Generation in Scientific Documents

TL;DR

This paper presents MLBCAP, a collaborative framework using multiple specialized large language models to generate high-quality, contextually rich figure captions for scientific documents, surpassing existing methods and even human performance.

Contribution

The paper introduces a novel multi-LLM framework that improves scientific figure captioning by data quality assessment, diverse caption generation, and a caption selection process, addressing limitations of prior single-model approaches.

Findings

Human evaluations favor MLBCAP-generated captions over human-written ones.

Filtering low-quality data improves captioning performance.

Collaborative LLMs produce more informative captions than existing methods.

Abstract

Scientific figure captioning is a complex task that requires generating contextually appropriate descriptions of visual content. However, existing methods often fall short by utilizing incomplete information, treating the task solely as either an image-to-text or text summarization problem. This limitation hinders the generation of high-quality captions that fully capture the necessary details. Moreover, existing data sourced from arXiv papers contain low-quality captions, posing significant challenges for training large language models (LLMs). In this paper, we introduce a framework called Multi-LLM Collaborative Figure Caption Generation (MLBCAP) to address these challenges by leveraging specialized LLMs for distinct sub-tasks. Our approach unfolds in three key modules: (Quality Assessment) We utilize multimodal LLMs to assess the quality of training data, enabling the filtration of low-quality captions. (Diverse Caption Generation) We then employ a strategy of fine-tuning/prompting multiple LLMs on the captioning task to generate candidate captions. (Judgment) Lastly, we prompt a prominent LLM to select the highest quality caption from the candidates, followed by refining any remaining inaccuracies. Human evaluations demonstrate that informative captions produced by our approach rank better than human-written captions, highlighting its effectiveness. Our code is available at https://github.com/teamreboott/MLBCAP