Improving Image Captioning by Concept-based Sentence Reranking

TL;DR

This paper presents a concept-based sentence reranking method that enhances image captioning models by leveraging concept annotations, leading to improved performance on the ImageCLEF 2015 benchmark.

Contribution

The paper introduces a black-box compatible reranking approach using concept detection to improve image captioning accuracy, outperforming previous methods.

Findings

Achieved a METEOR score of 0.1875 on ImageCLEF 2015 test set.

Outperformed the runner-up with a significant margin.

Demonstrated the effectiveness of concept-based reranking in image captioning.

Abstract

This paper describes our winning entry in the ImageCLEF 2015 image sentence generation task. We improve Google's CNN-LSTM model by introducing concept-based sentence reranking, a data-driven approach which exploits the large amounts of concept-level annotations on Flickr. Different from previous usage of concept detection that is tailored to specific image captioning models, the propose approach reranks predicted sentences in terms of their matches with detected concepts, essentially treating the underlying model as a black box. This property makes the approach applicable to a number of existing solutions. We also experiment with fine tuning on the deep language model, which improves the performance further. Scoring METEOR of 0.1875 on the ImageCLEF 2015 test set, our system outperforms the runner-up (METEOR of 0.1687) with a clear margin.