ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

TL;DR

This paper introduces ICBench, a comprehensive image captioning benchmark with diverse captions and an innovative ITIScore metric that aligns well with human judgments and generalizes across datasets.

Contribution

The paper presents a new large-scale benchmark dataset, ICBench, and an automated ITIScore metric for evaluating image captioning models, addressing limitations of existing benchmarks.

Findings

ICBench covers 12 content categories with 40K captions from 10 MLLMs.

ITIScore correlates strongly with human subjective scores.

ITIScore demonstrates robust zero-shot performance on other datasets.

Abstract

Recent advances in multimodal large language models (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing image captioning benchmarks typically suffer from limited diversity in caption length, the absence of recent advanced MLLMs, and insufficient human annotations, which potentially introduces bias and limits the ability to comprehensively assess the performance of modern MLLMs. To address these limitations, we present a new large-scale image captioning benchmark, termed, ICBench, which covers 12 content categories and consists of both short and long captions generated by 10 advanced MLLMs on 2K images, resulting in 40K captions in total. We conduct extensive human subjective studies to obtain mean opinion scores (MOSs) across fine-grained evaluation dimensions, where short captions are assessed in terms of fluency, relevance, and conciseness, while long captions are evaluated based on fluency, relevance, and completeness. Furthermore, we propose an automated evaluation metric, \textbf{ITIScore}, based on an image-to-text-to-image framework, which measures caption quality through reconstruction consistency. Experimental results demonstrate strong alignment between our automatic metric and human judgments, as well as robust zero-shot generalization ability on other public captioning datasets. Both the dataset and model will be released upon publication.