Uncertainty-Aware Image Captioning

TL;DR

This paper introduces an uncertainty-aware image captioning framework that iteratively inserts words based on their uncertainty levels, improving caption quality and decoding speed through a non-autoregressive hierarchy.

Contribution

It proposes a novel parallel and iterative captioning method that accounts for word uncertainty, enhancing explainability and efficiency over traditional sequential models.

Findings

Outperforms baseline in caption quality on MS COCO

Achieves faster decoding with logarithmic time complexity

Provides more explainable caption generation process

Abstract

It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.