Diverse and Coherent Paragraph Generation from Images

TL;DR

This paper introduces a novel method for generating diverse, coherent paragraphs from images by incorporating coherence and global topic vectors, using a variational auto-encoder, and demonstrates superior performance over existing methods.

Contribution

It proposes a new paragraph generation approach that enhances coherence and diversity using coherence vectors, global topic vectors, and a variational auto-encoder framework.

Findings

Outperforms state-of-the-art methods on two datasets

Produces more coherent and diverse paragraph descriptions

Effectively models ambiguity in image-to-paragraph mapping

Abstract

Paragraph generation from images, which has gained popularity recently, is an important task for video summarization, editing, and support of the disabled. Traditional image captioning methods fall short on this front, since they aren't designed to generate long informative descriptions. Moreover, the vanilla approach of simply concatenating multiple short sentences, possibly synthesized from a classical image captioning system, doesn't embrace the intricacies of paragraphs: coherent sentences, globally consistent structure, and diversity. To address those challenges, we propose to augment paragraph generation techniques with 'coherence vectors', 'global topic vectors', and modeling of the inherent ambiguity of associating paragraphs with images, via a variational auto-encoder formulation. We demonstrate the effectiveness of the developed approach on two datasets, outperforming existing state-of-the-art techniques on both.