Why Do Large Language Models Generate Harmful Content?

TL;DR

This paper uses causal mediation analysis to identify that harmful content in large language models mainly originates in later layers, especially within MLP blocks and specific neurons, revealing how harmful signals propagate through the model.

Contribution

It introduces a causal analysis framework to pinpoint the layers, modules, and neurons responsible for harmful content generation in LLMs, providing insights into the propagation of harmful signals.

Findings

Harmful content mainly arises in later layers of LLMs.

Failures in MLP blocks are the primary cause of harmful generation.

Specific neurons act as gating mechanisms for harmful content.

Abstract

Large Language Models (LLMs) have been shown to generate harmful content. However, the underlying causes of such behavior remain under explored. We propose a causal mediation analysis-based approach to identify the causal factors responsible for harmful generation. Our method performs a multi-granular analysis across model layers, modules (MLP and attention blocks), and individual neurons. Extensive experiments on state-of-the-art LLMs indicate that harmful generation arises in the later layers of the model, results primarily from failures in MLP blocks rather than attention blocks, and is associated with neurons that act as a gating mechanism for harmful generation. The results indicate that the early layers in the model are used for a contextual understanding of harmfulness in a prompt, which is then propagated through the model, to generate harmfulness in the late layers, as well as a signal indicating harmfulness through MLP blocks. This is then further propagated to the last layer of the model, specifically to a sparse set of neurons, which receives the signal and determines the generation of harmful content accordingly.