# Scene-based Factored Attention for Image Captioning

## TL;DR

This paper introduces a scene-based factored attention module for image captioning that leverages scene concepts to improve caption quality, outperforming existing methods on the Microsoft COCO benchmark.

## Contribution

It proposes a novel scene-based factored attention mechanism that explicitly incorporates scene concepts into image captioning models, enhancing their semantic understanding.

## Key findings

- Significant performance improvement on Microsoft COCO benchmark
- Outperforms state-of-the-art approaches across multiple metrics
- Effective integration of scene concepts into attention mechanisms

## Abstract

Image captioning has attracted ever-increasing research attention in the multimedia community. To this end, most cutting-edge works rely on an encoder-decoder framework with attention mechanisms, which have achieved remarkable progress. However, such a framework does not consider scene concepts to attend visual information, which leads to sentence bias in caption generation and defects the performance correspondingly. We argue that such scene concepts capture higher-level visual semantics and serve as an important cue in describing images. In this paper, we propose a novel scene-based factored attention module for image captioning. Specifically, the proposed module first embeds the scene concepts into factored weights explicitly and attends the visual information extracted from the input image. Then, an adaptive LSTM is used to generate captions for specific scene types. Experimental results on Microsoft COCO benchmark show that the proposed scene-based attention module improves model performance a lot, which outperforms the state-of-the-art approaches under various evaluation metrics.