Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs

TL;DR

This paper compares unimodal, multimodal, and generative models for social media event detection, highlighting the strengths and limitations of each approach and proposing new insights into their effectiveness.

Contribution

It introduces a comprehensive study of multimodal and generative models for social media event detection, revealing their comparative performance and challenges.

Findings

Multimodal approaches outperform unimodal methods.

Generative models lag behind supervised methods in precision.

Generative models handle social media text issues better.

Abstract

In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches.