Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

TL;DR

This paper enhances LSTM-based video description by integrating linguistic knowledge from large text corpora, leading to improved grammaticality and modest quality improvements in generated descriptions.

Contribution

It introduces a method to incorporate neural language models and distributional semantics into video description models, leveraging large text corpora for better language quality.

Findings

Significant improvement in grammaticality of descriptions

Modest enhancement in descriptive quality

Effective integration of linguistic knowledge into video captioning

Abstract

This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.