Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

TL;DR

This paper introduces Flab-Pruner, a unified structural pruning method for Code LLMs that reduces parameters by 22% while maintaining performance, leading to more efficient and environmentally friendly generative coding models.

Contribution

The paper presents a novel unified pruning approach combining vocabulary, layer, and FFN pruning tailored for generative Code LLMs, along with a data strategy to improve performance recovery.

Findings

Retains 97% of original performance after pruning 22% of parameters.

Achieves comparable or better performance post-pruning and post-training.

Significantly improves storage, GPU usage, and energy efficiency.

Abstract

The extensive application of Large Language Models (LLMs) in generative coding tasks has raised concerns due to their high computational demands and energy consumption. Unlike previous structural pruning methods designed for classification models that deal with lowdimensional classification logits, generative Code LLMs produce high-dimensional token logit sequences, making traditional pruning objectives inherently limited. Moreover, existing single component pruning approaches further constrain the effectiveness when applied to generative Code LLMs. In response, we propose Flab-Pruner, an innovative unified structural pruning method that combines vocabulary, layer, and Feed-Forward Network (FFN) pruning. This approach effectively reduces model parameters while maintaining performance. Additionally, we introduce a customized code instruction data strategy for coding tasks to enhance the performance recovery efficiency of the pruned model. Through extensive evaluations on three state-of-the-art Code LLMs across multiple generative coding tasks, the results demonstrate that Flab-Pruner retains 97% of the original performance after pruning 22% of the parameters and achieves the same or even better performance after post-training. The pruned models exhibit significant improvements in storage, GPU usage, computational efficiency, and environmental impact, while maintaining well robustness. Our research provides a sustainable solution for green software engineering and promotes the efficient deployment of LLMs in real-world generative coding intelligence applications.