Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation

TL;DR

This paper presents a semi-supervised knowledge distillation approach for classifying customer transactions from free text, achieving high accuracy with fewer parameters, and is deployed in production for trend analysis.

Contribution

It introduces a cost-effective semi-supervised framework using knowledge distillation with RoBERTa as teacher and ALBERT as student for transaction classification.

Findings

RoBERTa outperforms other models in categorization tasks.

ALBERT with knowledge distillation achieves comparable performance to RoBERTa.

The approach is effective on large internal and public datasets.

Abstract

In pickup and delivery services, transaction classification based on customer provided free text is a challenging problem. It involves the association of a wide variety of customer inputs to a fixed set of categories while adapting to the various customer writing styles. This categorization is important for the business: it helps understand the market needs and trends, and also assist in building a personalized experience for different segments of the customers. Hence, it is vital to capture these category information trends at scale, with high precision and recall. In this paper, we focus on a specific use-case where a single category drives each transaction. We propose a cost-effective transaction classification approach based on semi-supervision and knowledge distillation frameworks. The approach identifies the category of a transaction using free text input given by the customer. We use weak labelling and notice that the performance gains are similar to that of using human-annotated samples. On a large internal dataset and on 20Newsgroup dataset, we see that RoBERTa performs the best for the categorization tasks. Further, using an ALBERT model (it has 33x fewer parameters vis-a-vis parameters of RoBERTa), with RoBERTa as the Teacher, we see a performance similar to that of RoBERTa and better performance over unadapted ALBERT. This framework, with ALBERT as a student and RoBERTa as teacher, is further referred to as R-ALBERT in this paper. The model is in production and is used by business to understand changing trends and take appropriate decisions.