Neural Language Modeling with Visual Features

TL;DR

This paper enhances RNN language models by integrating visual features from videos, demonstrating significant perplexity improvements and providing insights into multimodal language modeling.

Contribution

It introduces a multimodal RNN language model with visual features, exploring architecture choices and achieving substantial perplexity reduction on large-scale datasets.

Findings

Middle fusion architecture yields best performance.

Over 25% relative perplexity improvement.

Insights into multimodal language model advantages.

Abstract

Multimodal language models attempt to incorporate non-linguistic features for the language modeling task. In this work, we extend a standard recurrent neural network (RNN) language model with features derived from videos. We train our models on data that is two orders-of-magnitude bigger than datasets used in prior work. We perform a thorough exploration of model architectures for combining visual and text features. Our experiments on two corpora (YouCookII and 20bn-something-something-v2) show that the best performing architecture consists of middle fusion of visual and text features, yielding over 25% relative improvement in perplexity. We report analysis that provides insights into why our multimodal language model improves upon a standard RNN language model.