Multimodal Classification of Events in Social Media

TL;DR

This paper explores the use of textual, visual, and combined multimodal data for classifying social media content related to events, demonstrating that multimodal approaches outperform existing methods.

Contribution

It provides an extensive analysis of different modalities and their combinations for social event classification, establishing a new baseline with improved results.

Findings

Multimodal representations outperform unimodal methods.

Textual and visual modalities have complementary strengths.

Experimental results set a new benchmark for future research.

Abstract

A large amount of social media hosted on platforms like Flickr and Instagram is related to social events. The task of social event classification refers to the distinction of event and non-event-related content as well as the classification of event types (e.g. sports events, concerts, etc.). In this paper, we provide an extensive study of textual, visual, as well as multimodal representations for social event classification. We investigate strengths and weaknesses of the modalities and study synergy effects between the modalities. Experimental results obtained with our multimodal representation outperform state-of-the-art methods and provide a new baseline for future research.