Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception

TL;DR

This paper introduces EDC, a method that leverages off-the-shelf visual specialists to incorporate detailed visual attributes into image captions, significantly enhancing their descriptive quality and visual understanding capabilities.

Contribution

The paper proposes a novel approach called EDC that integrates rich visual attributes from trained specialists into captions, improving multimodal perception beyond existing captioning methods.

Findings

Enhanced caption quality with detailed attributes

Improved performance on visual understanding tasks

Better reasoning capabilities in multimodal models

Abstract

Training Large Multimodality Models (LMMs) relies on descriptive image caption that connects image and language. Existing methods for generating such captions often rely on distilling the captions from pretrained LMMs, constructing them from publicly available internet images, or even generating them through human annotation. However, these strategies can fall short in terms of precision and granularity, particularly when dealing with complex visual reasoning tasks. In this paper, we propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption. Our approach, named EDC, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. By systematically integrating these rich attributes into the generated captions, EDC significantly improves the descriptive quality of the captions, providing a deeper and more nuanced understanding of the visual content. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. The complete source code of EDC pipeline and datasets will be available at https://github.com/syp2ysy/DCE.