Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models

TL;DR

This paper investigates how the distribution of harmful information varies across image encoder layers in vision-language models and introduces a layer-wise training method to improve safety alignment by reducing harmful outputs.

Contribution

It reveals the uneven distribution of harmful info in VLM layers and proposes Layer-Wise PPO to enhance safety alignment through layer-wise reinforcement learning.

Findings

Early exits increase harmful responses in VLMs.

Layer-Wise PPO reduces harmful outputs across datasets.

Layer-wise analysis informs safer model design.

Abstract

Vision-language models (VLMs) have improved significantly in their capabilities, but their complex architecture makes their safety alignment challenging. In this paper, we reveal an uneven distribution of harmful information across the intermediate layers of the image encoder and show that skipping a certain set of layers and exiting early can increase the chance of the VLM generating harmful responses. We call it as "Image enCoder Early-exiT" based vulnerability (ICET). Our experiments across three VLMs: LLaVA-1.5, LLaVA-NeXT, and Llama 3.2, show that performing early exits from the image encoder significantly increases the likelihood of generating harmful outputs. To tackle this, we propose a simple yet effective modification of the Clipped-Proximal Policy Optimization (Clip-PPO) algorithm for performing layer-wise multi-modal RLHF for VLMs. We term this as Layer-Wise PPO (L-PPO). We evaluate our L-PPO algorithm across three multimodal datasets and show that it consistently reduces the harmfulness caused by early exits.