A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews

TL;DR

This study compares traditional and transformer-based models for sentiment analysis of Bangladeshi government banking app reviews, revealing classical models outperform transformers and highlighting language resource gaps.

Contribution

It introduces a hybrid labeling approach and provides a comprehensive analysis of sentiment classification performance across models and languages.

Findings

Classical models like Random Forest achieved higher accuracy than transformers.

Transformers showed no significant advantage over traditional models in this context.

Significant performance gap exists between Bangla and English sentiment analysis accuracy.

Abstract

For millions of users in developing economies who depend on mobile banking as their primary gateway to financial services, app quality directly shapes financial access. The study analyzed 5,652 Google Play reviews in English and Bangla (filtered from 11,414 raw reviews) for four Bangladeshi government banking apps. The authors used a hybrid labeling approach that combined use of the reviewer's star rating for each review along with a separate independent XLM-RoBERTa classifier to produce moderate inter-method agreement (kappa = 0.459). Traditional models outperformed transformer-based ones: Random Forest produced the highest accuracy (0.815), while Linear SVM produced the highest weighted F1 score (0.804); both were higher than the performance of fine-tuned XLM-RoBERTa (0.793). McNemar's test confirmed that all classical models were significantly superior to the off-the-shelf XLM-RoBERTa (p < 0.05), while differences with the fine-tuned variant were not statistically significant. DeBERTa-v3 was applied to analyze the sentiment at the aspect level across the reviews for the four apps; the reviewers expressed their dissatisfaction primarily with the speed of transactions and with the poor design of interfaces; eJanata app received the worst ratings from the reviewers across all apps. Three policy recommendations are made based on these findings - remediation of app quality, trust-centred release management, and Bangla-first NLP adoption - to assist state-owned banks in moving towards improving their digital services through data-driven methods. Notably, a 16.1-percentage-point accuracy gap between Bangla and English text highlights the need for low-resource language model development.