Instruction-Following Pruning for Large Language Models

TL;DR

This paper introduces instruction-following pruning, a dynamic, input-dependent structured pruning method for large language models that adapts parameters based on user instructions, improving efficiency and performance.

Contribution

It proposes a novel dynamic pruning approach that uses a sparse mask predictor conditioned on user instructions, enhancing model efficiency and task adaptability.

Findings

3B activated model outperforms the 3B dense model by 5-8 points.

Achieves performance comparable to a 9B model on various benchmarks.

Demonstrates effectiveness across math and coding domains.

Abstract

With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.