Sparse Graph to Sequence Learning for Vision Conditioned Long Textual Sequence Generation

TL;DR

This paper introduces Sparse Graph-to-Sequence Transformer (SGST), a novel approach for generating long textual descriptions conditioned on visual data by encoding graph semantics and producing coherent stories, significantly improving over previous methods.

Contribution

The paper proposes a new Sparse Graph-to-Sequence Transformer model that encodes graph-level semantics for long text generation conditioned on visual content, advancing beyond traditional sentence-level methods.

Findings

Achieved 13.3% improvement on CIDEr score over state-of-the-art.

Effectively encodes graph semantics for long sequence generation.

Demonstrates strong performance on image paragraph dataset.

Abstract

Generating longer textual sequences when conditioned on the visual information is an interesting problem to explore. The challenge here proliferate over the standard vision conditioned sentence-level generation (e.g., image or video captioning) as it requires to produce a brief and coherent story describing the visual content. In this paper, we mask this Vision-to-Sequence as Graph-to-Sequence learning problem and approach it with the Transformer architecture. To be specific, we introduce Sparse Graph-to-Sequence Transformer (SGST) for encoding the graph and decoding a sequence. The encoder aims to directly encode graph-level semantics, while the decoder is used to generate longer sequences. Experiments conducted with the benchmark image paragraph dataset show that our proposed achieve 13.3% improvement on the CIDEr evaluation measure when comparing to the previous state-of-the-art approach.