Generating Accurate and Detailed Captions for High-Resolution Images

TL;DR

This paper introduces a multi-stage pipeline combining vision-language models, large language models, and object detection to generate more accurate, detailed, and hallucination-free captions for high-resolution images.

Contribution

It presents a novel multi-stage process that refines captions by integrating object detection and language models, improving caption detail and reducing hallucinations for high-res images.

Findings

Enhanced captions with more detailed object descriptions

Reduced hallucinations in generated captions

Improved evaluation scores on high-resolution image datasets

Abstract

Vision-language models (VLMs) often struggle to generate accurate and detailed captions for high-resolution images since they are typically pre-trained on low-resolution inputs (e.g., 224x224 or 336x336 pixels). Downscaling high-resolution images to these dimensions may result in the loss of visual details and the omission of important objects. To address this limitation, we propose a novel pipeline that integrates vision-language models, large language models (LLMs), and object detection systems to enhance caption quality. Our proposed pipeline refines captions through a novel, multi-stage process. Given a high-resolution image, an initial caption is first generated using a VLM, and key objects in the image are then identified by an LLM. The LLM predicts additional objects likely to co-occur with the identified key objects, and these predictions are verified by object detection systems. Newly detected objects not mentioned in the initial caption undergo focused, region-specific captioning to ensure they are incorporated. This process enriches caption detail while reducing hallucinations by removing references to undetected objects. We evaluate the enhanced captions using pairwise comparison and quantitative scoring from large multimodal models, along with a benchmark for hallucination detection. Experiments on a curated dataset of high-resolution images demonstrate that our pipeline produces more detailed and reliable image captions while effectively minimizing hallucinations.