Learning Contextualized Semantics from Co-occurring Terms via a Siamese Architecture

TL;DR

This paper introduces a novel Siamese architecture for learning contextualized semantics from co-occurring descriptive terms, effectively handling out-of-vocabulary concepts and improving semantic priming across multimedia domains.

Contribution

The paper proposes a new Siamese architecture that models contextualized semantics from descriptive terms using pattern aggregation and probabilistic topic models, enabling better concept embeddings.

Findings

Outperforms state-of-the-art methods in semantic priming

Effectively represents OOV concepts in embedding space

Demonstrates robustness across multiple multimedia datasets

Abstract

One of the biggest challenges in Multimedia information retrieval and understanding is to bridge the semantic gap by properly modeling concept semantics in context. The presence of out of vocabulary (OOV) concepts exacerbates this difficulty. To address the semantic gap issues, we formulate a problem on learning contextualized semantics from descriptive terms and propose a novel Siamese architecture to model the contextualized semantics from descriptive terms. By means of pattern aggregation and probabilistic topic models, our Siamese architecture captures contextualized semantics from the co-occurring descriptive terms via unsupervised learning, which leads to a concept embedding space of the terms in context. Furthermore, the co-occurring OOV concepts can be easily represented in the learnt concept embedding space. The main properties of the concept embedding space are demonstrated via visualization. Using various settings in semantic priming, we have carried out a thorough evaluation by comparing our approach to a number of state-of-the-art methods on six annotation corpora in different domains, i.e., MagTag5K, CAL500 and Million Song Dataset in the music domain as well as Corel5K, LabelMe and SUNDatabase in the image domain. Experimental results on semantic priming suggest that our approach outperforms those state-of-the-art methods considerably in various aspects.