A CNN-RNN Framework for Image Annotation from Visual Cues and Social Network Metadata

TL;DR

This paper introduces a CNN-RNN framework that combines visual features and social media metadata to improve image annotation, especially for unclear or uncommon images, by leveraging context and semantic embeddings.

Contribution

The paper proposes a novel CNN-RNN model that jointly utilizes visual cues and social media metadata, enhancing robustness to vocabulary changes and decoupling from low-level metadata representations.

Findings

Outperforms state-of-the-art models on NUS-WIDE dataset

Reduces sensory and semantic gaps in image annotation

Demonstrates robustness to vocabulary variations

Abstract

Images represent a commonly used form of visual communication among people. Nevertheless, image classification may be a challenging task when dealing with unclear or non-common images needing more context to be correctly annotated. Metadata accompanying images on social-media represent an ideal source of additional information for retrieving proper neighborhoods easing image annotation task. To this end, we blend visual features extracted from neighbors and their metadata to jointly leverage context and visual cues. Our models use multiple semantic embeddings to achieve the dual objective of being robust to vocabulary changes between train and test sets and decoupling the architecture from the low-level metadata representation. Convolutional and recurrent neural networks (CNNs-RNNs) are jointly adopted to infer similarity among neighbors and query images. We perform comprehensive experiments on the NUS-WIDE dataset showing that our models outperform state-of-the-art architectures based on images and metadata, and decrease both sensory and semantic gaps to better annotate images.