Not-in-Perspective: Towards Shielding Google's Perspective API Against Adversarial Negation Attacks

TL;DR

This paper introduces formal reasoning techniques to enhance Google's Perspective API, making it more resilient against adversarial negation attacks in toxicity detection on social media comments.

Contribution

It proposes a novel formal reasoning wrapper that improves toxicity detection accuracy by defending against negation-based adversarial attacks, complementing existing machine learning models.

Findings

Hybrid methods outperform purely statistical solutions.

Significant accuracy improvements against negation attacks.

Effective integration of formal reasoning with ML models.

Abstract

The rise of cyberbullying in social media platforms involving toxic comments has escalated the need for effective ways to monitor and moderate online interactions. Existing solutions of automated toxicity detection systems, are based on a machine or deep learning algorithms. However, statistics-based solutions are generally prone to adversarial attacks that contain logic based modifications such as negation in phrases and sentences. In that regard, we present a set of formal reasoning-based methodologies that wrap around existing machine learning toxicity detection systems. Acting as both pre-processing and post-processing steps, our formal reasoning wrapper helps alleviating the negation attack problems and significantly improves the accuracy and efficacy of toxicity scoring. We evaluate different variations of our wrapper on multiple machine learning models against a negation adversarial dataset. Experimental results highlight the improvement of hybrid (formal reasoning and machine-learning) methods against various purely statistical solutions.