Visual Semantic Relatedness Dataset for Image Captioning

TL;DR

This paper introduces a new dataset that extends COCO Captions with scene-related textual information, enabling better integration of NLP techniques into image captioning systems.

Contribution

It provides a textual visual context dataset that enhances image captioning by incorporating scene information, bridging computer vision and NLP.

Findings

Enables use of NLP methods for captioning tasks

Facilitates end-to-end training or post-processing approaches

Improves semantic understanding in image captioning

Abstract

Modern image captioning system relies heavily on extracting knowledge from images to capture the concept of a static story. In this paper, we propose a textual visual context dataset for captioning, in which the publicly available dataset COCO Captions (Lin et al., 2014) has been extended with information about the scene (such as objects in the image). Since this information has a textual form, it can be used to leverage any NLP task, such as text similarity or semantic relation methods, into captioning systems, either as an end-to-end training strategy or a post-processing based approach.