FullAnno: A Data Engine for Enhancing Image Comprehension of MLLMs

TL;DR

FullAnno is a scalable data engine that generates high-quality, detailed image annotations to improve multimodal large language models' understanding and reasoning in vision-language tasks.

Contribution

The paper introduces FullAnno, a novel cascade annotation system that re-annotates datasets with richer, more detailed image labels to enhance MLLMs' performance.

Findings

Re-annotated COCO and Visual Genome datasets with increased object and caption details.

Enhanced LLaVA-v1.5 performance on multiple benchmarks using FullAnno data.

Tripled object annotations and 15-fold longer captions compared to original datasets.

Abstract

Multimodal Large Language Models (MLLMs) have shown promise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they heavily depend on high-quality data in the Supervised Fine-Tuning (SFT) phase. The existing approaches aim to curate high-quality data via GPT-4V, but they are not scalable due to the commercial nature of GPT-4V and the simplicity of the prompts used to instruct the model. To this end, we devised the FullAnno system, which is a data engine that can generate large-scale, high-quality, and fine-grained image annotations consisting of the category and position of objects, region descriptions, text information, as well as image dense captions. This engine is characterized by its cascade annotation process, which involves multiple expert models and employs rich prompts to instruct LLMs in generating dense image captions. We re-annotated the COCO and Visual Genome datasets using our FullAnno system, tripling the number of object annotations and increasing the length of the original image captions by a factor of 15. Experiments show that the regenerated annotation can significantly enhance the capabilities of LLaVA-v1.5 on several benchmarks. The re-annotated data are available at: https://arcana-project-page.github.io