Unitary Multi-Margin BERT for Robust Natural Language Processing

TL;DR

This paper introduces UniBERT, a robust NLP model combining unitary weights and multi-margin loss, significantly enhancing resistance to adversarial attacks while maintaining accuracy and offering adjustable robustness parameters.

Contribution

The paper proposes UniBERT, a novel adversarial defense method for BERT that is computationally efficient and improves robustness through the integration of unitary weights and multi-margin loss.

Findings

Post-attack accuracy improved by up to 73.8%.

Pre-attack accuracy remains competitive.

Robustness tradeoff adjustable via a scalar parameter.

Abstract

Recent developments in adversarial attacks on deep learning leave many mission-critical natural language processing (NLP) systems at risk of exploitation. To address the lack of computationally efficient adversarial defense methods, this paper reports a novel, universal technique that drastically improves the robustness of Bidirectional Encoder Representations from Transformers (BERT) by combining the unitary weights with the multi-margin loss. We discover that the marriage of these two simple ideas amplifies the protection against malicious interference. Our model, the unitary multi-margin BERT (UniBERT), boosts post-attack classification accuracies significantly by 5.3% to 73.8% while maintaining competitive pre-attack accuracies. Furthermore, the pre-attack and post-attack accuracy tradeoff can be adjusted via a single scalar parameter to best fit the design requirements for the target applications.