New Directions in Text Classification Research: Maximizing The Performance of Sentiment Classification from Limited Data

TL;DR

This paper addresses the challenge of sentiment classification with limited training data by providing benchmarks, external data, and optimized SVM-based scores, aiming to improve accuracy in real-world scenarios.

Contribution

It introduces a benchmark dataset and baseline scores for sentiment classification with limited data, along with external data for augmentation and a focus on optimizing SVM performance.

Findings

Baseline F1-score: 40.83%

Optimized F1-score: 51.28%

External data improves classification performance

Abstract

The stakeholders' needs in sentiment analysis for various issues, whether positive or negative, are speed and accuracy. One new challenge in sentiment analysis tasks is the limited training data, which often leads to suboptimal machine learning models and poor performance on test data. This paper discusses the problem of text classification based on limited training data (300 to 600 samples) into three classes: positive, negative, and neutral. A benchmark dataset is provided for training and testing data on the issue of Kaesang Pangarep's appointment as Chairman of PSI. External data for aggregation and augmentation purposes are provided, consisting of two datasets: the topic of Covid Vaccination sentiment and an open topic. The official score used is the F1-score, which balances precision and recall among the three classes, positive, negative, and neutral. A baseline score is provided as a reference for researchers for unoptimized classification methods. The optimized score is provided as a reference for the target score to be achieved by any proposed method. Both scoring (baseline and optimized) use the SVM method, which is widely reported as the state-of-the-art in conventional machine learning methods. The F1-scores achieved by the baseline and optimized methods are 40.83% and 51.28%, respectively.