Jointly Modeling Embedding and Translation to Bridge Video and Language

TL;DR

This paper introduces LSTM-E, a unified framework that jointly models visual content and sentence semantics to improve automatic video description, achieving state-of-the-art results on the YouTube2Text dataset.

Contribution

The paper proposes a novel joint embedding and LSTM framework that simultaneously learns visual-semantic relationships and sequence generation for video captioning.

Findings

Achieves 45.3% BLEU@4 and 31.0% METEOR on YouTube2Text.

Outperforms existing methods in SVO triplet prediction.

Demonstrates the effectiveness of joint embedding in video captioning.

Abstract

Automatically describing video content with natural language is a fundamental challenge of multimedia. Recurrent Neural Networks (RNN), which models sequence dynamics, has attracted increasing attention on visual interpretation. However, most existing approaches generate a word locally with given previous words and the visual content, while the relationship between sentence semantics and visual content is not holistically exploited. As a result, the generated sentences may be contextually correct but the semantics (e.g., subjects, verbs or objects) are not true. This paper presents a novel unified framework, named Long Short-Term Memory with visual-semantic Embedding (LSTM-E), which can simultaneously explore the learning of LSTM and visual-semantic embedding. The former aims to locally maximize the probability of generating the next word given previous words and visual content, while the latter is to create a visual-semantic embedding space for enforcing the relationship between the semantics of the entire sentence and visual content. Our proposed LSTM-E consists of three components: a 2-D and/or 3-D deep convolutional neural networks for learning powerful video representation, a deep RNN for generating sentences, and a joint embedding model for exploring the relationships between visual content and sentence semantics. The experiments on YouTube2Text dataset show that our proposed LSTM-E achieves to-date the best reported performance in generating natural sentences: 45.3% and 31.0% in terms of BLEU@4 and METEOR, respectively. We also demonstrate that LSTM-E is superior in predicting Subject-Verb-Object (SVO) triplets to several state-of-the-art techniques.