Panoptic Captioning: An Equivalence Bridge for Image and Text

TL;DR

This paper introduces panoptic captioning, a comprehensive image description task, and proposes new methods and metrics to improve and evaluate model performance, demonstrating significant advancements over existing large language models.

Contribution

It formulates panoptic captioning as a novel task, develops PancapEngine and PancapChain for data generation and stepwise captioning, and introduces PancapScore for evaluation.

Findings

PancapChain-13B outperforms state-of-the-art open-source models.

Our methods surpass proprietary models like GPT-4o and Gemini-2.0-Pro.

The proposed data engine and method significantly improve panoptic captioning performance.

Abstract

This work introduces panoptic captioning, a novel task striving to seek the minimum text equivalent of images, which has broad potential applications. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method. Project page: https://visual-ai.github.io/pancap/