IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method

TL;DR

IM-BERT enhances BERT's robustness against adversarial attacks by modeling its layers as solutions to ODEs and applying the implicit Euler method, leading to significant performance improvements without extra parameters.

Contribution

This paper introduces IM-BERT, a novel approach that conceptualizes BERT layers as ODE solutions and applies the implicit Euler method to improve robustness against adversarial attacks.

Findings

IM-BERT outperforms BERT by 8.3% on AdvGLUE.

IM-BERT achieves 5.9% higher accuracy in low-resource scenarios.

The implicit Euler approach enhances numerical stability and robustness.

Abstract

Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: the explicit and implicit Euler approaches. Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT's layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3\%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9\%p higher accuracy.