ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models

TL;DR

This paper introduces ShufText, a black box method to evaluate text classification models' reliance on keywords by shuffling words and assessing accuracy, revealing many models' limited semantic understanding.

Contribution

The paper presents ShufText, a simple technique to expose over-reliance on keywords in text classifiers, highlighting their limited semantic comprehension and robustness issues.

Findings

Models often maintain accuracy after word shuffling, indicating reliance on keywords.

Pretraining with language models does not significantly improve semantic understanding.

Both simple and complex models show limited robustness to out-of-domain sentences.

Abstract

Text classification is the most basic natural language processing task. It has a wide range of applications ranging from sentiment analysis to topic classification. Recently, deep learning approaches based on CNN, LSTM, and Transformers have been the de facto approach for text classification. In this work, we highlight a common issue associated with these approaches. We show that these systems are over-reliant on the important words present in the text that are useful for classification. With limited training data and discriminative training strategy, these approaches tend to ignore the semantic meaning of the sentence and rather just focus on keywords or important n-grams. We propose a simple black box technique ShutText to present the shortcomings of the model and identify the over-reliance of the model on keywords. This involves randomly shuffling the words in a sentence and evaluating the classification accuracy. We see that on common text classification datasets there is very little effect of shuffling and with high probability these models predict the original class. We also evaluate the effect of language model pretraining on these models and try to answer questions around model robustness to out of domain sentences. We show that simple models based on CNN or LSTM as well as complex models like BERT are questionable in terms of their syntactic and semantic understanding.