Injecting Prior Knowledge into Image Caption Generation

TL;DR

This paper introduces a novel approach to enhance image captioning by integrating prior knowledge through latent topics and regularization, resulting in more accurate and human-like captions, especially in low-data scenarios.

Contribution

It proposes a new method combining latent topic attention and regularization to improve image captioning performance and interpretability.

Findings

Significant improvement on MSCOCO dataset

Enhanced caption interpretability

Better performance in low-data regimes

Abstract

Automatically generating natural language descriptions from an image is a challenging problem in artificial intelligence that requires a good understanding of the visual and textual signals and the correlations between them. The state-of-the-art methods in image captioning struggles to approach human level performance, especially when data is limited. In this paper, we propose to improve the performance of the state-of-the-art image captioning models by incorporating two sources of prior knowledge: (i) a conditional latent topic attention, that uses a set of latent variables (topics) as an anchor to generate highly probable words and, (ii) a regularization technique that exploits the inductive biases in syntactic and semantic structure of captions and improves the generalization of image captioning models. Our experiments validate that our method produces more human interpretable captions and also leads to significant improvements on the MSCOCO dataset in both the full and low data regimes.