Combining Language and Vision with a Multimodal Skip-gram Model

TL;DR

This paper introduces a multimodal skip-gram model that integrates visual information with linguistic context to improve semantic understanding and zero-shot image labeling, revealing visual properties of abstract words.

Contribution

The paper presents MMSKIP-GRAM, a novel model that combines text and visual data for word representation, enhancing semantic tasks and zero-shot image retrieval.

Findings

Achieves strong performance on semantic benchmarks

Improves zero-shot image labeling and retrieval

Discovers visual properties of abstract words

Abstract

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.