Effective Black Box Testing of Sentiment Analysis Classification Networks

TL;DR

This paper introduces a black-box testing approach for transformer-based sentiment analysis models, using input space partitioning and emotional feature coverage to improve test comprehensiveness and identify model vulnerabilities.

Contribution

It proposes novel coverage criteria and a k-projection coverage metric for systematic testing of sentiment analysis networks, leveraging large language models for test case generation.

Findings

Average 16% increase in test coverage

Average 6.5% decrease in model accuracy

Effective identification of model vulnerabilities

Abstract

Transformer-based neural networks have demonstrated remarkable performance in natural language processing tasks such as sentiment analysis. Nevertheless, the issue of ensuring the dependability of these complicated architectures through comprehensive testing is still open. This paper presents a collection of coverage criteria specifically designed to assess test suites created for transformer-based sentiment analysis networks. Our approach utilizes input space partitioning, a black-box method, by considering emotionally relevant linguistic features such as verbs, adjectives, adverbs, and nouns. In order to effectively produce test cases that encompass a wide range of emotional elements, we utilize the k-projection coverage metric. This metric minimizes the complexity of the problem by examining subsets of k features at the same time, hence reducing dimensionality. Large language models are employed to generate sentences that display specific combinations of emotional features. The findings from experiments obtained from a sentiment analysis dataset illustrate that our criteria and generated tests have led to an average increase of 16\% in test coverage. In addition, there is a corresponding average decrease of 6.5\% in model accuracy, showing the ability to identify vulnerabilities. Our work provides a foundation for improving the dependability of transformer-based sentiment analysis systems through comprehensive test evaluation.