Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers

TL;DR

This paper introduces Interleaved Ensemble Unlearning (IEU), a novel finetuning method for Vision Transformers that effectively defends against backdoor attacks by filtering poisoned data through a two-stage process.

Contribution

The paper proposes a new unlearning-based defense method specifically designed for Vision Transformers, addressing the gap in backdoor defenses for ViTs.

Findings

IEU outperforms existing defenses against 11 backdoor attacks

It is effective across multiple datasets and model architectures

IEU maintains model accuracy while removing backdoors

Abstract

Vision Transformers (ViTs) have become popular in computer vision tasks. Backdoor attacks, which trigger undesirable behaviours in models during inference, threaten ViTs' performance, particularly in security-sensitive tasks. Although backdoor defences have been developed for Convolutional Neural Networks (CNNs), they are less effective for ViTs, and defences tailored to ViTs are scarce. To address this, we present Interleaved Ensemble Unlearning (IEU), a method for finetuning clean ViTs on backdoored datasets. In stage 1, a shallow ViT is finetuned to have high confidence on backdoored data and low confidence on clean data. In stage 2, the shallow ViT acts as a ``gate'' to block potentially poisoned data from the defended ViT. This data is added to an unlearn set and asynchronously unlearned via gradient ascent. We demonstrate IEU's effectiveness on three datasets against 11 state-of-the-art backdoor attacks and show its versatility by applying it to different model architectures.