Modeling the Severity of Complaints in Social Media

TL;DR

This paper introduces a computational approach to classify complaint severity levels in social media, utilizing transformer models and linguistic features, achieving state-of-the-art results in complaint detection and severity prediction.

Contribution

It is the first to model complaint severity levels computationally, enriching a dataset and applying multi-task learning with transformer networks for improved accuracy.

Findings

Achieved 55.7 macro F1 in complaint severity classification.

Reached 88.2 macro F1 in binary complaint detection.

Provided qualitative insights into model behavior.

Abstract

The speech act of complaining is used by humans to communicate a negative mismatch between reality and expectations as a reaction to an unfavorable situation. Linguistic theory of pragmatics categorizes complaints into various severity levels based on the face-threat that the complainer is willing to undertake. This is particularly useful for understanding the intent of complainers and how humans develop suitable apology strategies. In this paper, we study the severity level of complaints for the first time in computational linguistics. To facilitate this, we enrich a publicly available data set of complaints with four severity categories and train different transformer-based networks combined with linguistic information achieving 55.7 macro F1. We also jointly model binary complaint classification and complaint severity in a multi-task setting achieving new state-of-the-art results on binary complaint detection reaching up to 88.2 macro F1. Finally, we present a qualitative analysis of the behavior of our models in predicting complaint severity levels.