CaptionQA: Is Your Caption as Useful as the Image Itself?

TL;DR

CaptionQA introduces a comprehensive benchmark to evaluate how well image captions support downstream tasks across multiple domains, revealing significant gaps in current model capabilities.

Contribution

It presents a new utility-based benchmark with extensive annotations and questions to assess caption usefulness in real-world applications.

Findings

State-of-the-art models show up to 32% lower utility in caption-based tasks compared to image-based tasks.

CaptionQA covers 4 domains with 33,027 annotated questions, enabling detailed utility evaluation.

Models perform substantially worse on caption utility than on traditional image-QA benchmarks.

Abstract

Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.