A Hybrid Model for Combining Neural Image Caption and k-Nearest Neighbor Approach for Image Captioning

TL;DR

This paper introduces a hybrid image captioning model that combines neural and k-nearest neighbor methods, using a logistic regression classifier to select the best caption, resulting in improved BLEU-4 scores on Flickr8k.

Contribution

The paper presents a novel hybrid approach that integrates NIC and k-NN models with a classifier to enhance image captioning performance.

Findings

Hybrid model outperforms individual models in BLEU-4 score

BLEU-4 score of 18.20 on Flickr8k dataset

Effective combination of neural and k-NN methods

Abstract

A hybrid model is proposed that integrates two popular image captioning methods to generate a text-based summary describing the contents of the image. The two image captioning models are the Neural Image Caption (NIC) and the k-nearest neighbor approach. These are trained individually on the training set. We extract a set of five features, from the validation set, for evaluating the results of the two models that in turn is used to train a logistic regression classifier. The BLEU-4 scores of the two models are compared for generating the binary-value ground truth for the logistic regression classifier. For the test set, the input images are first passed separately through the two models to generate the individual captions. The five-dimensional feature set extracted from the two models is passed to the logistic regression classifier to take a decision regarding the final caption generated which is the best of two captions generated by the models. Our implementation of the k-nearest neighbor model achieves a BLEU-4 score of 15.95 and the NIC model achieves a BLEU-4 score of 16.01, on the benchmark Flickr8k dataset. The proposed hybrid model is able to achieve a BLEU-4 score of 18.20 proving the validity of our approach.