A Study of the Attention Abnormality in Trojaned BERTs

TL;DR

This paper analyzes how Trojaned BERT models exhibit abnormal attention focus behavior upon encountering poisoned inputs and introduces an attention-based detector to identify such Trojaned models, providing new insights into Trojan mechanisms.

Contribution

It is the first to analyze the Trojan mechanism in BERT through attention focus behavior and develop a detector based on this insight.

Findings

Trojaned BERTs show attention focus drifting towards trigger tokens.

The proposed detector effectively distinguishes Trojaned models from clean ones.

The study offers new understanding of Trojan attack mechanisms in transformer models.

Abstract

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention.