HARRISON: A Benchmark on HAshtag Recommendation for Real-world Images in Social Networks

TL;DR

This paper introduces HARRISON, a new benchmark dataset for hashtag recommendation on real-world social media images, and evaluates baseline models to promote further research in image-based hashtag prediction.

Contribution

The paper presents the first vision-only benchmark dataset for hashtag recommendation on social media images and provides baseline models for future research.

Findings

Baseline models demonstrate the importance of contextual understanding.

Object-based and scene-based models perform differently on the dataset.

Integrated models show improved performance over single feature models.

Abstract

Simple, short, and compact hashtags cover a wide range of information on social networks. Although many works in the field of natural language processing (NLP) have demonstrated the importance of hashtag recommendation, hashtag recommendation for images has barely been studied. In this paper, we introduce the HARRISON dataset, a benchmark on hashtag recommendation for real world images in social networks. The HARRISON dataset is a realistic dataset, composed of 57,383 photos from Instagram and an average of 4.5 associated hashtags for each photo. To evaluate our dataset, we design a baseline framework consisting of visual feature extractor based on convolutional neural network (CNN) and multi-label classifier based on neural network. Based on this framework, two single feature-based models, object-based and scene-based model, and an integrated model of them are evaluated on the HARRISON dataset. Our dataset shows that hashtag recommendation task requires a wide and contextual understanding of the situation conveyed in the image. As far as we know, this work is the first vision-only attempt at hashtag recommendation for real world images in social networks. We expect this benchmark to accelerate the advancement of hashtag recommendation.