Explain Images with Multimodal Recurrent Neural Networks

TL;DR

This paper introduces a multimodal RNN model that generates descriptive sentences for images by integrating deep neural networks for both image and language understanding, validated on multiple datasets.

Contribution

The novel multimodal RNN architecture effectively combines image and sentence modeling, outperforming existing methods in image captioning and retrieval tasks.

Findings

Outperforms state-of-the-art in image captioning on benchmark datasets

Achieves significant improvements in image and sentence retrieval tasks

Validates effectiveness across three diverse datasets

Abstract

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel sentence descriptions to explain the content of images. It directly models the probability distribution of generating a word given previous words and the image. Image descriptions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on three benchmark datasets (IAPR TC-12, Flickr 8K, and Flickr 30K). Our model outperforms the state-of-the-art generative method. In addition, the m-RNN model can be applied to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.