Analyzing Multi-Head Attention on Trojan BERT Models

TL;DR

This paper examines how multi-head attention functions differently in Trojan-infected BERT models compared to benign ones, identifying specific attention heads associated with malicious behavior.

Contribution

It characterizes attention head functions in Trojan BERT models, revealing specific 'trojan' heads responsible for malicious misclassifications.

Findings

Identification of specific attention heads linked to trojan behavior

Trojan models perform normally on clean inputs but misclassify triggered inputs

Analysis of attention head functions differentiates benign and trojan models

Abstract

This project investigates the behavior of multi-head attention in Transformer models, specifically focusing on the differences between benign and trojan models in the context of sentiment analysis. Trojan attacks cause models to perform normally on clean inputs but exhibit misclassifications when presented with inputs containing predefined triggers. We characterize attention head functions in trojan and benign models, identifying specific 'trojan' heads and analyzing their behavior.