LEAP: Efficient and Automated Test Method for NLP Software

TL;DR

LEAP is an automated testing method for NLP software that uses Levy flight-based optimization to generate adversarial test cases, significantly improving success rates and efficiency over existing methods.

Contribution

This paper introduces LEAP, a novel test method combining Levy flight and adaptive particle swarm optimization for more effective and faster adversarial test case generation in NLP.

Findings

LEAP achieves a 79.1% success rate in generating adversarial test cases.

LEAP reduces testing time by up to 147.6 seconds compared to other methods.

LEAP enhances the robustness of DNN-based NLP systems.

Abstract

The widespread adoption of DNNs in NLP software has highlighted the need for robustness. Researchers proposed various automatic testing techniques for adversarial test cases. However, existing methods suffer from two limitations: weak error-discovering capabilities, with success rates ranging from 0% to 24.6% for BERT-based NLP software, and time inefficiency, taking 177.8s to 205.28s per test case, making them challenging for time-constrained scenarios. To address these issues, this paper proposes LEAP, an automated test method that uses LEvy flight-based Adaptive Particle swarm optimization integrated with textual features to generate adversarial test cases. Specifically, we adopt Levy flight for population initialization to increase the diversity of generated test cases. We also design an inertial weight adaptive update operator to improve the efficiency of LEAP's global optimization of high-dimensional text examples and a mutation operator based on the greedy strategy to reduce the search time. We conducted a series of experiments to validate LEAP's ability to test NLP software and found that the average success rate of LEAP in generating adversarial test cases is 79.1%, which is 6.1% higher than the next best approach (PSOattack). While ensuring high success rates, LEAP significantly reduces time overhead by up to 147.6s compared to other heuristic-based methods. Additionally, the experimental results demonstrate that LEAP can generate more transferable test cases and significantly enhance the robustness of DNN-based systems.