INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models

TL;DR

INTERLACE introduces a layer pruning and adaptation framework for large vision-language models that maintains high performance with minimal data and computational resources by interleaving pruning, finetuning, and freezing layers.

Contribution

It proposes a novel interleaved layer pruning and adaptation method that preserves performance in large VLMs with minimal finetuning and data.

Findings

Achieves 88.9% performance retention after removing 25% of layers.

Requires only 1% of the dataset for finetuning.

Outperforms existing pruning methods in VLMs.

Abstract

We introduce INTERLACE, a novel framework that prunes redundant layers in VLMs while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lead to significant performance drop when applied to VLMs. Instead, we analyze triplets of consecutive layers to identify local redundancy, removing the most redundant of the first two layers, finetune the remaining layer to compensate for the lost capacity, and freeze the third layer to serve as a stable anchor during finetuning. We found that this interleaved finetune-freeze design enables rapid convergence with minimal data after pruning. By finetuning only a subset of layers on just 1% of the FineVision dataset for one epoch, Interlace achieves 88.9% average performance retention after dropping 25% of the network, achieving SOTA performance. Our code is available at: https://github.com/pmadinei/Interlace.git