Fusing Text and Image for Event Detection in Twitter

TL;DR

This paper presents a novel method for event detection in Twitter streams that combines textual and visual data, significantly improving accuracy over single-modality approaches.

Contribution

It introduces a fusion approach that integrates text and image analysis for more accurate event detection in social media streams.

Findings

Achieved 94% accuracy in event detection.

Outperformed text-only and image-only methods.

Demonstrated effectiveness of multimodal data fusion.

Abstract

In this contribution, we develop an accurate and effective event detection method to detect events from a Twitter stream, which uses visual and textual information to improve the performance of the mining process. The method monitors a Twitter stream to pick up tweets having texts and images and stores them into a database. This is followed by applying a mining algorithm to detect an event. The procedure starts with detecting events based on text only by using the feature of the bag-of-words which is calculated using the term frequency-inverse document frequency (TF-IDF) method. Then it detects the event based on image only by using visual features including histogram of oriented gradients (HOG) descriptors, grey-level cooccurrence matrix (GLCM), and color histogram. K nearest neighbours (Knn) classification is used in the detection. The final decision of the event detection is made based on the reliabilities of text only detection and image only detection. The experiment result showed that the proposed method achieved high accuracy of 0.94, comparing with 0.89 with texts only, and 0.86 with images only.