TextAttack: Lessons learned in designing Python frameworks for NLP

TL;DR

TextAttack is an open-source Python toolkit that consolidates NLP adversarial attack research, enabling researchers and developers to evaluate and improve NLP models efficiently across various datasets and frameworks.

Contribution

It unifies multiple NLP adversarial attack methods into a single, well-documented framework, addressing key challenges in usability and compatibility.

Findings

Successfully integrated 15+ papers into one framework

Facilitated testing of NLP model weaknesses across datasets

Provided insights for developing robust NLP Python tools

Abstract

TextAttack is an open-source Python toolkit for adversarial attacks, adversarial training, and data augmentation in NLP. TextAttack unites 15+ papers from the NLP adversarial attack literature into a single framework, with many components reused across attacks. This framework allows both researchers and developers to test and study the weaknesses of their NLP models. To build such an open-source NLP toolkit requires solving some common problems: How do we enable users to supply models from different deep learning frameworks? How can we build tools to support as many different datasets as possible? We share our insights into developing a well-written, well-documented NLP Python framework in hope that they can aid future development of similar packages.