Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script

TL;DR

This paper presents HITL-GAT, an interactive system for generating adversarial texts with human input, specifically applied to Tibetan script, to improve NLP robustness in low-resource languages.

Contribution

We introduce HITL-GAT, a novel human-in-the-loop system for adversarial text generation tailored to low-resource languages, and establish the first Tibetan adversarial robustness benchmark.

Findings

HITL-GAT effectively generates high-quality adversarial texts for Tibetan.

The system enhances robustness evaluation for low-resource languages.

Benchmark results reveal vulnerabilities in Tibetan language models.

Abstract

DNN-based language models excel across various NLP tasks but remain highly vulnerable to textual adversarial attacks. While adversarial text generation is crucial for NLP security, explainability, evaluation, and data augmentation, related work remains overwhelmingly English-centric, leaving the problem of constructing high-quality and sustainable adversarial robustness benchmarks for lower-resourced languages both difficult and understudied. First, method customization for lower-resourced languages is complicated due to linguistic differences and limited resources. Second, automated attacks are prone to generating invalid or ambiguous adversarial texts. Last but not least, language models continuously evolve and may be immune to parts of previously generated adversarial texts. To address these challenges, we introduce HITL-GAT, an interactive system based on a general approach to human-in-the-loop generation of adversarial texts. Additionally, we demonstrate the utility of HITL-GAT through a case study on Tibetan script, employing three customized adversarial text generation methods and establishing its first adversarial robustness benchmark, providing a valuable reference for other lower-resourced languages.