Sentiment Analysis of Social Media Data for Predicting Consumer Behavior Trends Using Machine Learning

TL;DR

This paper demonstrates how advanced machine learning models, especially BERT and LSTM, can analyze Twitter sentiment data to accurately predict consumer behavior trends and identify evolving public opinions.

Contribution

It introduces a comprehensive workflow for sentiment analysis on social media data, comparing multiple models and addressing challenges like sarcasm detection and multilingual processing.

Findings

BERT achieved the highest classification accuracy of 83.48%.

LSTM and BERT effectively captured linguistic and contextual patterns.

Temporal sentiment shifts and key themes were identified through analysis.

Abstract

In the era of rapid technological advancement, social media platforms such as Twitter (X) have emerged as indispensable tools for gathering consumer insights, capturing diverse opinions, and understanding public attitudes. This research applies advanced machine learning methods for sentiment analysis on Twitter data, with a focus on predicting consumer trends. Using the Sentiment140 dataset, the study detects evolving patterns in consumer preferences with "car" as an example. A structured workflow was used to clean and prepare data for analysis. Machine learning models, including Support Vector Machines (SVM), Naive Bayes, Long Short-Term Memory (LSTM) networks, and Bidirectional Encoder Representations from Transformers (BERT), were employed to classify sentiments and predict trends. Model performance was measured using accuracy, precision, recall, and F1 score, with BERT achieving the highest results (Accuracy: 83.48%, Precision: 79.37%, Recall: 90.60%, F1: 84.61). Results show that LSTM and BERT effectively capture linguistic and contextual patterns, improving prediction accuracy and providing insights into consumer behavior. Temporal analysis revealed sentiment shifts across time, while Named Entity Recognition (NER) identified related terms and themes. This research addresses challenges like sarcasm detection and multilingual data processing, offering a scalable framework for generating actionable consumer insights.