If You Don't Understand It, Don't Use It: Eliminating Trojans with Filters Between Layers

TL;DR

This paper proposes a practical filtering approach, including LoRA filters, to eliminate unknown trojans in large language models, focusing on empirical effectiveness and insights into model information processing.

Contribution

Introduces a general filtering method and specific LoRA-based implementation to unlearn trojans in LLMs, addressing a critical security challenge.

Findings

Filters are most effective on residual streams and recent layers.

The approach works on small to medium models.

Some behaviors suggest fundamental questions about LLM information storage.

Abstract

Large language models (LLMs) sometimes exhibit dangerous unintended behaviors. Finding and fixing these is challenging because the attack surface is massive -- it is not tractable to exhaustively search for all possible inputs that may elicit such behavior. One specific and particularly challenging case is that if data-poisoning-injected trojans, since there is no way to know what they are to search for them. To our knowledge, there is no generally applicable method to unlearn unknown trojans injected during pre-training. This work seeks to provide a general purpose recipe (filters) and a specific implementation (LoRA) filters that work in practice on small to medium sized models. The focus is primarily empirical, though some perplexing behavior opens the door to the fundamental question of how LLMs store and process information. Not unexpectedly, we find that our filters work best on the residual stream and the latest layers.