Deep Neural Networks In Fully Connected CRF For Image Labeling With Social Network Metadata

TL;DR

This paper introduces a novel deep learning framework combining CNNs and RNNs within a fully connected CRF to improve image labeling by integrating visual content with social media metadata, outperforming existing methods.

Contribution

The paper presents a new CRF-based model that fuses image content with social network metadata using deep neural networks, with CRF modeled as an RNN for efficient learning and inference.

Findings

Outperforms state-of-the-art methods on MIR-9K dataset

Effectively combines visual and social media information

Handles data imbalance with weighted ranking and cross entropy losses

Abstract

We propose a novel method for predicting image labels by fusing image content descriptors with the social media context of each image. An image uploaded to a social media site such as Flickr often has meaningful, associated information, such as comments and other images the user has uploaded, that is complementary to pixel content and helpful in predicting labels. Prediction challenges such as ImageNet~\cite{imagenet_cvpr09} and MSCOCO~\cite{LinMBHPRDZ:ECCV14} use only pixels, while other methods make predictions purely from social media context \cite{McAuleyECCV12}. Our method is based on a novel fully connected Conditional Random Field (CRF) framework, where each node is an image, and consists of two deep Convolutional Neural Networks (CNN) and one Recurrent Neural Network (RNN) that model both textual and visual node/image information. The edge weights of the CRF graph represent textual similarity and link-based metadata such as user sets and image groups. We model the CRF as an RNN for both learning and inference, and incorporate the weighted ranking loss and cross entropy loss into the CRF parameter optimization to handle the training data imbalance issue. Our proposed approach is evaluated on the MIR-9K dataset and experimentally outperforms current state-of-the-art approaches.