Toxicity Detection towards Adaptability to Changing Perturbations

TL;DR

This paper introduces a new dataset and benchmark for toxicity detection that evaluates models' robustness against evolving perturbation patterns, proposing a continual learning approach to improve adaptability to new malicious tactics.

Contribution

It presents a novel dataset with diverse perturbation patterns, systematically evaluates existing methods' vulnerabilities, and proposes a domain incremental learning paradigm for enhanced robustness.

Findings

Current methods are vulnerable to new perturbation patterns.

The proposed continual learning approach improves detection robustness.

Benchmark results highlight the need for adaptive toxicity detection models.

Abstract

Toxicity detection is crucial for maintaining the peace of the society. While existing methods perform well on normal toxic contents or those generated by specific perturbation methods, they are vulnerable to evolving perturbation patterns. However, in real-world scenarios, malicious users tend to create new perturbation patterns for fooling the detectors. For example, some users may circumvent the detector of large language models (LLMs) by adding `I am a scientist' at the beginning of the prompt. In this paper, we introduce a novel problem, i.e., continual learning jailbreak perturbation patterns, into the toxicity detection field. To tackle this problem, we first construct a new dataset generated by 9 types of perturbation patterns, 7 of them are summarized from prior work and 2 of them are developed by us. We then systematically validate the vulnerability of current methods on this new perturbation pattern-aware dataset via both the zero-shot and fine tuned cross-pattern detection. Upon this, we present the domain incremental learning paradigm and the corresponding benchmark to ensure the detector's robustness to dynamically emerging types of perturbed toxic text. Our code and dataset are provided in the appendix and will be publicly available at GitHub, by which we wish to offer new research opportunities for the security-relevant communities.