Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Models

TL;DR

This paper introduces Amnesia, an adversarial attack method that manipulates internal states of large language models to bypass safety measures, revealing vulnerabilities in current safety mechanisms and emphasizing the need for more robust defenses.

Contribution

The study presents a novel activation-space adversarial attack called Amnesia that effectively bypasses existing safety safeguards in open-weight LLMs without additional training.

Findings

Amnesia successfully circumvents safety mechanisms in state-of-the-art LLMs.

The attack induces antisocial behaviors in models on benchmark datasets.

Current safety measures are insufficient against internal activation manipulations.

Abstract

Warning: This article includes red-teaming experiments, which contain examples of compromised LLM responses that may be offensive or upsetting. Large Language Models (LLMs) have the potential to create harmful content, such as generating sophisticated phishing emails and assisting in writing code of harmful computer viruses. Thus, it is crucial to ensure their safe and responsible response generation. To reduce the risk of generating harmful or irresponsible content, researchers have developed techniques such as reinforcement learning with human feedback to align LLM's outputs with human values and preferences. However, it is still undetermined whether such measures are sufficient to prevent LLMs from generating interesting responses. In this study, we propose Amnesia, a lightweight activation-space adversarial attack that manipulates internal transformer states to bypass existing safety mechanisms in open-weight LLMs. Through experimental analysis on state-of-the-art, open-weight LLMs, we demonstrate that our attack effectively circumvents existing safeguards, enabling the generation of harmful content without the need for any fine-tuning or additional training. Our experiments on benchmark datasets show that the proposed attack can induce various antisocial behaviors in LLMs. These findings highlight the urgent need for more robust security measures in open-weight LLMs and underscore the importance of continued research to prevent their potential misuse.