A Dataset and Benchmarks for Multimedia Social Analysis

TL;DR

This paper introduces a large, multi-modal social media dataset with images, videos, and text, enabling advances in multimedia social analysis through various tasks and baseline performance evaluations.

Contribution

The paper provides a new extensive multi-modal social media dataset with detailed statistics and baseline benchmarks for multiple multimedia analysis tasks.

Findings

Dataset contains 677k posts and millions of images, videos, and comments.

Baseline models achieve measurable performance on regression tasks.

The dataset supports diverse applications like captioning, classification, and sentiment analysis.

Abstract

We present a new publicly available dataset with the goal of advancing multi-modality learning by offering vision and language data within the same context. This is achieved by obtaining data from a social media website with posts containing multiple paired images/videos and text, along with comment trees containing images/videos and/or text. With a total of 677k posts, 2.9 million post images, 488k post videos, 1.4 million comment images, 4.6 million comment videos, and 96.9 million comments, data from different modalities can be jointly used to improve performances for a variety of tasks such as image captioning, image classification, next frame prediction, sentiment analysis, and language modeling. We present a wide range of statistics for our dataset. Finally, we provide baseline performance analysis for one of the regression tasks using pre-trained models and several fully connected networks.