Customer Sentiment Analysis using Weak Supervision for Customer-Agent Chat

TL;DR

This paper develops a weak supervision approach to sentiment analysis in customer-agent chat data, demonstrating improved domain-specific performance over existing APIs by leveraging domain-specific rules and fine-tuning RoBERTa.

Contribution

It introduces a method combining weak sentiment classifiers and lexicon-based rules to train an effective customer chat sentiment classifier, outperforming off-the-shelf NLP APIs.

Findings

Weak supervision with labeling functions improves sentiment classification accuracy.

Domain-specific knowledge enhances model performance over generic APIs.

Customer sentiment correlates with problem resolution in chat interactions.

Abstract

Prior work on sentiment analysis using weak supervision primarily focuses on different reviews such as movies (IMDB), restaurants (Yelp), products (Amazon).~One under-explored field in this regard is customer chat data for a customer-agent chat in customer support due to the lack of availability of free public data. Here, we perform sentiment analysis on customer chat using weak supervision on our in-house dataset. We fine-tune the pre-trained language model (LM) RoBERTa as a sentiment classifier using weak supervision. Our contribution is as follows:1) We show that by using weak sentiment classifiers along with domain-specific lexicon-based rules as Labeling Functions (LF), we can train a fairly accurate customer chat sentiment classifier using weak supervision. 2) We compare the performance of our custom-trained model with off-the-shelf google cloud NLP API for sentiment analysis. We show that by injecting domain-specific knowledge using LFs, even with weak supervision, we can train a model to handle some domain-specific use cases better than off-the-shelf google cloud NLP API. 3) We also present an analysis of how customer sentiment in a chat relates to problem resolution.