Performance Evaluation of Sentiment Analysis on Text and Emoji Data Using End-to-End, Transfer Learning, Distributed and Explainable AI Models

TL;DR

This study evaluates sentiment analysis models on text and emoji data, comparing end-to-end, transfer learning, distributed training, and explainability techniques, revealing high accuracy on text but challenges with unseen emojis.

Contribution

It introduces a comprehensive approach combining sentence embeddings, neural networks, distributed training, and explainability for sentiment analysis on text and emojis.

Findings

Text sentiment classification accuracy ~98%

Emoji accuracy drops to 70% with unseen emojis

Distributed training reduces runtime by 15%

Abstract

Emojis are being frequently used in todays digital world to express from simple to complex thoughts more than ever before. Hence, they are also being used in sentiment analysis and targeted marketing campaigns. In this work, we performed sentiment analysis of Tweets as well as on emoji dataset from the Kaggle. Since tweets are sentences we have used Universal Sentence Encoder (USE) and Sentence Bidirectional Encoder Representations from Transformers (SBERT) end-to-end sentence embedding models to generate the embeddings which are used to train the Standard fully connected Neural Networks (NN), and LSTM NN models. We observe the text classification accuracy was almost the same for both the models around 98 percent. On the contrary, when the validation set was built using emojis that were not present in the training set then the accuracy of both the models reduced drastically to 70 percent. In addition, the models were also trained using the distributed training approach instead of a traditional singlethreaded model for better scalability. Using the distributed training approach, we were able to reduce the run-time by roughly 15% without compromising on accuracy. Finally, as part of explainable AI the Shap algorithm was used to explain the model behaviour and check for model biases for the given feature set.