OPCap:Object-aware Prompting Captioning

TL;DR

OPCap introduces an object-aware prompting method that leverages object detection and attribute refinement to reduce hallucination and improve image captioning quality, validated on COCO and nocaps datasets.

Contribution

The paper presents a novel target-aware prompting strategy that integrates object labels and attributes to enhance captioning accuracy and reduce hallucination.

Findings

Significant reduction in object hallucination in captions.

Improved caption quality on COCO and nocaps datasets.

Effective integration of object detection and attribute prediction.

Abstract

In the field of image captioning, the phenomenon where missing or nonexistent objects are used to explain an image is referred to as object bias (or hallucination). To mitigate this issue, we propose a target-aware prompting strategy. This method first extracts object labels and their spatial information from the image using an object detector. Then, an attribute predictor further refines the semantic features of the objects. These refined features are subsequently integrated and fed into the decoder, enhancing the model's understanding of the image context. Experimental results on the COCO and nocaps datasets demonstrate that OPCap effectively mitigates hallucination and significantly improves the quality of generated captions.