Alignment-Constrained Dynamic Pruning for LLMs: Identifying and Preserving Alignment-Critical Circuits

TL;DR

This paper introduces Alignment-Aware Probe Pruning (AAPP), a dynamic pruning method that adaptively preserves alignment-critical circuits in large language models, significantly reducing alignment degradation during inference.

Contribution

The paper proposes a novel dynamic structured pruning technique, AAPP, that maintains alignment-critical circuits to improve safety and efficiency in LLM deployment.

Findings

AAPP reduces refusal rates by 50% at matched compute.

AAPP improves alignment preservation during dynamic pruning.

Experiments on multiple LLMs demonstrate effectiveness.

Abstract

Large Language Models require substantial computational resources for inference, posing deployment challenges. While dynamic pruning offers superior efficiency over static methods through adaptive circuit selection, it exacerbates alignment degradation by retaining only input-dependent safety-critical circuit preservation across diverse inputs. As a result, addressing these heightened alignment vulnerabilities remains critical. We introduce Alignment-Aware Probe Pruning (AAPP), a dynamic structured pruning method that adaptively preserves alignment-relevant circuits during inference, building upon Probe Pruning. Experiments on LLaMA 2-7B, Qwen2.5-14B-Instruct, and Gemma-3-12B-IT show AAPP improves refusal rates by 50\% at matched compute, enabling efficient yet safety-preserving LLM deployment.