Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning

TL;DR

This paper introduces a hierarchical-topic-guided framework for image paragraph captioning that integrates semantic topics with visual features to produce coherent, diverse, and interpretable descriptions, outperforming many existing methods.

Contribution

It develops a plug-and-play model combining deep topic modeling with visual extraction and language generation, enabling semantic coherence and interpretability in image paragraph captioning.

Findings

Competitive performance on public datasets

Ability to distill interpretable semantic topics

Generation of diverse, coherent captions

Abstract

Observing a set of images and their corresponding paragraph-captions, a challenging task is to learn how to produce a semantically coherent paragraph to describe the visual content of an image. Inspired by recent successes in integrating semantic topics into this task, this paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework, which couples a visual extractor with a deep topic model to guide the learning of a language model. To capture the correlations between the image and text at multiple levels of abstraction and learn the semantic topics from images, we design a variational inference network to build the mapping from image features to textual captions. To guide the paragraph generation, the learned hierarchical topics and visual features are integrated into the language model, including Long Short-Term Memory (LSTM) and Transformer, and jointly optimized. Experiments on public datasets demonstrate that the proposed models, which are competitive with many state-of-the-art approaches in terms of standard evaluation metrics, can be used to both distill interpretable multi-layer semantic topics and generate diverse and coherent captions. We release our code at https://github.com/DandanGuo1993/VTCM-based-image-paragraph-caption.git