Quality Assurance of A GPT-based Sentiment Analysis System: Adversarial Review Data Generation and Detection

TL;DR

This paper presents a method for quality assurance of a GPT-based sentiment analysis system by generating adversarial review data and detecting anomalies using surprise adequacy techniques, demonstrating effectiveness on Amazon reviews.

Contribution

It introduces a novel approach combining adversarial data generation and SA-based detection for LLM quality assurance in sentiment analysis.

Findings

Effective adversarial review data generation method

SA-based techniques successfully detect abnormal data

Improved understanding of LLM data quality issues

Abstract

Large Language Models (LLMs) have been garnering significant attention of AI researchers, especially following the widespread popularity of ChatGPT. However, due to LLMs' intricate architecture and vast parameters, several concerns and challenges regarding their quality assurance require to be addressed. In this paper, a fine-tuned GPT-based sentiment analysis model is first constructed and studied as the reference in AI quality analysis. Then, the quality analysis related to data adequacy is implemented, including employing the content-based approach to generate reasonable adversarial review comments as the wrongly-annotated data, and developing surprise adequacy (SA)-based techniques to detect these abnormal data. Experiments based on Amazon.com review data and a fine-tuned GPT model were implemented. Results were thoroughly discussed from the perspective of AI quality assurance to present the quality analysis of an LLM model on generated adversarial textual data and the effectiveness of using SA on anomaly detection in data quality assurance.