Robustness Tests of NLP Machine Learning Models: Search and Semantically Replace

TL;DR

This paper introduces a Search and Semantically Replace strategy to evaluate the robustness of NLP machine learning models by identifying important text parts and replacing them with semantically similar words, assessing model stability.

Contribution

It presents a novel framework for robustness testing in NLP models using targeted text modifications and compares different model types under this approach.

Findings

The strategy effectively identifies model vulnerabilities.

Different text representations show varying robustness levels.

The framework is adaptable to multiple NLP models.

Abstract

This paper proposes a strategy to assess the robustness of different machine learning models that involve natural language processing (NLP). The overall approach relies upon a Search and Semantically Replace strategy that consists of two steps: (1) Search, which identifies important parts in the text; (2) Semantically Replace, which finds replacements for the important parts, and constrains the replaced tokens with semantically similar words. We introduce different types of Search and Semantically Replace methods designed specifically for particular types of machine learning models. We also investigate the effectiveness of this strategy and provide a general framework to assess a variety of machine learning models. Finally, an empirical comparison is provided of robustness performance among three different model types, each with a different text representation.