Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue

TL;DR

This paper demonstrates that bootstrapping methods can significantly improve classifiers for detecting sarcasm and nastiness in online dialogue, addressing challenges posed by social media's conversational nature.

Contribution

It adapts and tests a bootstrapping approach from monologic NLP to dialogic contexts, enhancing sarcasm and nastiness detection with improved precision and recall.

Findings

Best sarcasm classifier: 62% precision, 52% recall.

Nastiness classifier: 75% precision, 62% recall.

Bootstrapping improves classifier performance in social media dialogue.

Abstract

More and more of the information on the web is dialogic, from Facebook newsfeeds, to forum conversations, to comment threads on news articles. In contrast to traditional, monologic Natural Language Processing resources such as news, highly social dialogue is frequent in social media, making it a challenging context for NLP. This paper tests a bootstrapping method, originally proposed in a monologic domain, to train classifiers to identify two different types of subjective language in dialogue: sarcasm and nastiness. We explore two methods of developing linguistic indicators to be used in a first level classifier aimed at maximizing precision at the expense of recall. The best performing classifier for the first phase achieves 54% precision and 38% recall for sarcastic utterances. We then use general syntactic patterns from previous work to create more general sarcasm indicators, improving precision to 62% and recall to 52%. To further test the generality of the method, we then apply it to bootstrapping a classifier for nastiness dialogic acts. Our first phase, using crowdsourced nasty indicators, achieves 58% precision and 49% recall, which increases to 75% precision and 62% recall when we bootstrap over the first level with generalized syntactic patterns.