Image Captioning with Semantic Attention

TL;DR

This paper introduces a semantic attention model for image captioning that combines top-down and bottom-up approaches, leading to significant improvements over existing methods on benchmark datasets.

Contribution

It proposes a novel semantic attention mechanism that fuses top-down and bottom-up processes for more accurate image captioning.

Findings

Outperforms state-of-the-art methods on COCO and Flickr30K datasets

Significantly improves evaluation metrics across benchmarks

Demonstrates effective integration of semantic concepts in caption generation

Abstract

Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics.