Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage

TL;DR

This paper introduces a multiagent approach and new evaluation metrics to improve the factual accuracy and coverage of hyper-detailed image captions generated by multimodal large language models, addressing hallucination issues.

Contribution

It proposes a collaborative multiagent method for caption correction and introduces a benchmark dataset with evaluation metrics tailored for detailed caption factuality.

Findings

The new evaluation method aligns better with human judgments of factuality.

The proposed approach significantly improves caption factual accuracy, even for GPT-4V.

VQA benchmark performance does not necessarily reflect captioning quality.

Abstract

Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MLLM factuality may fall short in hyper-detailed image captioning tasks. In contrast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions. Our code and data are available at https://github.com/adobe-research/CapMAS.