Semi-Autoregressive Image Captioning

TL;DR

The paper introduces SAIC, a semi-autoregressive image captioning framework that balances performance and speed by combining autoregressive and non-autoregressive methods, achieving faster inference with competitive accuracy.

Contribution

Proposes a novel two-stage semi-autoregressive model that improves captioning speed while maintaining high quality, bridging the gap between autoregressive and non-autoregressive approaches.

Findings

Outperforms previous non-autoregressive models on MS COCO

Achieves significant inference speedup

Maintains competitive caption quality

Abstract

Current state-of-the-art approaches for image captioning typically adopt an autoregressive manner, i.e., generating descriptions word by word, which suffers from slow decoding issue and becomes a bottleneck in real-time applications. Non-autoregressive image captioning with continuous iterative refinement, which eliminates the sequential dependence in a sentence generation, can achieve comparable performance to the autoregressive counterparts with a considerable acceleration. Nevertheless, based on a well-designed experiment, we empirically proved that iteration times can be effectively reduced when providing sufficient prior knowledge for the language decoder. Towards that end, we propose a novel two-stage framework, referred to as Semi-Autoregressive Image Captioning (SAIC), to make a better trade-off between performance and speed. The proposed SAIC model maintains autoregressive property in global but relieves it in local. Specifically, SAIC model first jumpily generates an intermittent sequence in an autoregressive manner, that is, it predicts the first word in every word group in order. Then, with the help of the partially deterministic prior information and image features, SAIC model non-autoregressively fills all the skipped words with one iteration. Experimental results on the MS COCO benchmark demonstrate that our SAIC model outperforms the preceding non-autoregressive image captioning models while obtaining a competitive inference speedup. Code is available at https://github.com/feizc/SAIC.