Uncovering Safety Risks of Large Language Models through Concept Activation Vector

TL;DR

This paper introduces the SCAV framework to interpret and attack large language models' safety mechanisms, revealing significant safety risks and transferability of attacks across models.

Contribution

We propose a novel Safety Concept Activation Vector (SCAV) framework and an SCAV-guided attack method to improve attack success rates and interpret LLM safety mechanisms.

Findings

Attack success rate of 99.14% on seven open-source LLMs

Generated attack prompts transfer to GPT-4

Embedding-level attacks transfer to other white-box LLMs

Abstract

Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, in our evaluation of seven open-source LLMs, we observe an average attack success rate of 99.14%, based on the classic keyword-matching criterion. Finally, we provide insights into the safety mechanism of LLMs. The code is available at https://github.com/SproutNan/AI-Safety_SCAV.