Cross Modification Attention Based Deliberation Model for Image Captioning

TL;DR

This paper introduces a two-pass image captioning framework with a novel Cross Modification Attention module, improving caption quality by refining drafts and leveraging future context, outperforming single-pass models.

Contribution

It proposes a universal two-pass decoding framework with CMA for enhanced semantic understanding and error correction in image captioning, a novel approach compared to traditional single-pass models.

Findings

Significant performance improvements over single-pass baselines.

Competitive results with state-of-the-art two-pass methods.

Effective error filtering and semantic enhancement via CMA.

Abstract

The conventional encoder-decoder framework for image captioning generally adopts a single-pass decoding process, which predicts the target descriptive sentence word by word in temporal order. Despite the great success of this framework, it still suffers from two serious disadvantages. Firstly, it is unable to correct the mistakes in the predicted words, which may mislead the subsequent prediction and result in error accumulation problem. Secondly, such a framework can only leverage the already generated words but not the possible future words, and thus lacks the ability of global planning on linguistic information. To overcome these limitations, we explore a universal two-pass decoding framework, where a single-pass decoding based model serving as the Drafting Model first generates a draft caption according to an input image, and a Deliberation Model then performs the polishing process to refine the draft caption to a better image description. Furthermore, inspired from the complementarity between different modalities, we propose a novel Cross Modification Attention (CMA) module to enhance the semantic expression of the image features and filter out error information from the draft captions. We integrate CMA with the decoder of our Deliberation Model and name it as Cross Modification Attention based Deliberation Model (CMA-DM). We train our proposed framework by jointly optimizing all trainable components from scratch with a trade-off coefficient. Experiments on MS COCO dataset demonstrate that our approach obtains significant improvements over single-pass decoding baselines and achieves competitive performances compared with other state-of-the-art two-pass decoding based methods.