Image Embedding Sampling Method for Diverse Captioning

TL;DR

This paper presents a training-free image captioning method that improves diversity and detail by focusing on distinct image regions using a small VLM, achieving performance comparable to larger models without extra training.

Contribution

It introduces a novel, training-free framework utilizing structured segmentation to enhance caption diversity and detail in small VLMs for image captioning.

Findings

Achieved high Div-2 scores on MSCOCO, Flickr30k, and Nocaps datasets.

Maintained strong image-caption relevancy and semantic integrity.

Enabled smaller VLMs to match larger models' performance without additional training.

Abstract

Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies. Alternatively, comparably smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image. In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone. Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics. Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity. We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset, respectively, while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions.