Backdoor Cleaning without External Guidance in MLLM Fine-tuning

TL;DR

This paper introduces BYE, a novel self-supervised framework that detects and filters backdoor samples in multimodal large language models by analyzing attention entropy patterns, without requiring external supervision.

Contribution

It presents a new backdoor defense method that leverages attention entropy analysis to identify malicious samples during fine-tuning of MLLMs without needing clean data or model modifications.

Findings

BYE achieves near-zero attack success rates in experiments.

It maintains high performance on clean tasks.

The method is effective across various datasets, models, and trigger types.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed in fine-tuning-as-a-service (FTaaS) settings, where user-submitted datasets adapt general-purpose models to downstream tasks. This flexibility, however, introduces serious security risks, as malicious fine-tuning can implant backdoors into MLLMs with minimal effort. In this paper, we observe that backdoor triggers systematically disrupt cross-modal processing by causing abnormal attention concentration on non-semantic regions--a phenomenon we term attention collapse. Based on this insight, we propose Believe Your Eyes (BYE), a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples. BYE operates via a three-stage pipeline: (1) extracting attention maps using the fine-tuned model, (2) computing entropy scores and profiling sensitive layers via bimodal separation, and (3) performing unsupervised clustering to remove suspicious samples. Unlike prior defenses, BYE equires no clean supervision, auxiliary labels, or model modifications. Extensive experiments across various datasets, models, and diverse trigger types validate BYE's effectiveness: it achieves near-zero attack success rates while maintaining clean-task performance, offering a robust and generalizable solution against backdoor threats in MLLMs.