HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models

TL;DR

HiPP-Prune is a hierarchical, preference-conditioned structured pruning framework for vision-language models that optimizes multiple objectives, including task utility and hallucination robustness, through plan-level decisions and a user-adjustable trade-off.

Contribution

It introduces a novel hierarchical pruning approach with a preference-conditioned policy and visual sensitivity integration, enabling controllable robustness-utility trade-offs in VLM pruning.

Findings

Discoveries of diverse non-dominated pruning plans.

Enhanced robustness-utility trade-offs demonstrated on LLaVA and ScienceQA.

Effective control over hallucination and task performance balance.

Abstract

Pruning vision-language models (VLMs) for efficient deployment is challenging because compression can affect not only task utility but also visual grounding, often amplifying object hallucinations even at the same sparsity level. We present HiPP-Prune, a hierarchical preference-conditioned structured pruning framework that treats pruning as conditional resource allocation under multiple objectives. HiPP-Prune makes plan-level decisions: a single policy invocation outputs a global pruning blueprint by factorizing decisions into an overall sparsity budget and a layer-wise allocation, enabling queryable trade-offs via a user-specified preference vector. To account for VLM-specific failure modes, our policy state integrates a visual sensitivity signal derived from attention flow between vision tokens and language hidden states, discouraging over-pruning of vision-critical layers that facilitate cross-modal fusion. We optimize pruning plans with plan-level Group Relative Policy Optimization (GRPO) under a multi-objective return that combines task utility, hallucination robustness (POPE), compression, and a synaptic-flow-inspired stability proxy to reduce unproductive exploration in high-sparsity regimes. Experiments on LLaVA with POPE and ScienceQA demonstrate that HiPP-Prune discovers diverse non-dominated pruning plans and provides controllable robustness--utility trade-offs under matched sparsity budgets.