Context and Attribute Grounded Dense Captioning

TL;DR

This paper introduces a novel dense captioning framework that incorporates multi-scale contextual reasoning and attribute grounding, significantly improving caption quality and contextual coherence in dense image descriptions.

Contribution

The work proposes an end-to-end framework with a contextual visual mining module and attribute grounded description generation, enhancing dense captioning with contextual and attribute-aware reasoning.

Findings

Outperforms state-of-the-art methods on Visual Genome dataset

Improves contextual coherence of generated captions

Enhances caption distinctiveness with hierarchical attribute supervision

Abstract

Dense captioning aims at simultaneously localizing semantic regions and describing these regions-of-interest (ROIs) with short phrases or sentences in natural language. Previous studies have shown remarkable progresses, but they are often vulnerable to the aperture problem that a caption generated by the features inside one ROI lacks contextual coherence with its surrounding context in the input image. In this work, we investigate contextual reasoning based on multi-scale message propagations from the neighboring contents to the target ROIs. To this end, we design a novel end-to-end context and attribute grounded dense captioning framework consisting of 1) a contextual visual mining module and 2) a multi-level attribute grounded description generation module. Knowing that captions often co-occur with the linguistic attributes (such as who, what and where), we also incorporate an auxiliary supervision from hierarchical linguistic attributes to augment the distinctiveness of the learned captions. Extensive experiments and ablation studies on Visual Genome dataset demonstrate the superiority of the proposed model in comparison to state-of-the-art methods.