Pix2Cap-COCO: Advancing Visual Comprehension via Pixel-Level Captioning

TL;DR

Pix2Cap-COCO introduces a pixel-level caption dataset and a new panoptic segmentation-captioning task, enabling models to generate detailed, instance-specific descriptions for fine-grained visual understanding.

Contribution

The paper presents the first panoptic pixel-level caption dataset and a novel task, along with a baseline model and demonstrates improved performance of large multimodal models through supervised fine-tuning.

Findings

Pix2Cap-COCO contains 167,254 detailed captions with an average of 22.94 words.

The dataset is challenging, requiring fine-grained visual and language understanding.

Fine-tuning with Pix2Cap-COCO improves model performance on multiple benchmarks.

Abstract

We present Pix2Cap-COCO, the first panoptic pixel-level caption dataset designed to advance fine-grained visual understanding. To achieve this, we carefully design an automated annotation pipeline that prompts GPT-4V to generate pixel-aligned, instance-specific captions for individual objects within images, enabling models to learn more granular relationships between objects and their contexts. This approach results in 167,254 detailed captions, with an average of 22.94 words per caption. Building on Pix2Cap-COCO, we introduce a novel task, panoptic segmentation-captioning, which challenges models to recognize instances in an image and provide detailed descriptions for each simultaneously. To benchmark this task, we design a robust baseline based on X-Decoder. The experimental results demonstrate that Pix2Cap-COCO is a particularly challenging dataset, as it requires models to excel in both fine-grained visual understanding and detailed language generation. Furthermore, we leverage Pix2Cap-COCO for Supervised Fine-Tuning (SFT) on large multimodal models (LMMs) to enhance their performance. For example, training with Pix2Cap-COCO significantly improves the performance of GPT4RoI, yielding gains in CIDEr +1.4%, ROUGE +0.4%, and SPICE +0.5% on Visual Genome dataset, and strengthens its region understanding ability on the ViP-BENCH, with an overall improvement of +5.1%, including notable increases in recognition accuracy +11.2% and language generation quality +22.2%.