Masked Non-Autoregressive Image Captioning

TL;DR

This paper introduces masked non-autoregressive decoding for image captioning, enabling parallel caption generation that improves diversity and semantic preservation over traditional autoregressive methods.

Contribution

It proposes a novel masked non-autoregressive decoding approach that addresses issues of slow generation and lack of diversity in captioning models.

Findings

More diverse caption generation

Better semantic content preservation

Faster inference compared to autoregressive models

Abstract

Existing captioning models often adopt the encoder-decoder architecture, where the decoder uses autoregressive decoding to generate captions, such that each token is generated sequentially given the preceding generated tokens. However, autoregressive decoding results in issues such as sequential error accumulation, slow generation, improper semantics and lack of diversity. Non-autoregressive decoding has been proposed to tackle slow generation for neural machine translation but suffers from multimodality problem due to the indirect modeling of the target distribution. In this paper, we propose masked non-autoregressive decoding to tackle the issues of both autoregressive decoding and non-autoregressive decoding. In masked non-autoregressive decoding, we mask several kinds of ratios of the input sequences during training, and generate captions parallelly in several stages from a totally masked sequence to a totally non-masked sequence in a compositional manner during inference. Experimentally our proposed model can preserve semantic content more effectively and can generate more diverse captions.