On the Compression of Language Models for Code: An Empirical Study on CodeBERT

TL;DR

This paper empirically examines how three compression strategies—knowledge distillation, quantization, and pruning—affect the performance and efficiency of CodeBERT across various software engineering tasks, providing insights for informed model optimization.

Contribution

It provides a comprehensive empirical analysis of the effects of common compression techniques on CodeBERT's performance in different software engineering tasks.

Findings

Impact varies by task and compression method

Compression can reduce inference latency and memory usage

Trade-offs between efficiency and effectiveness are task-dependent

Abstract

Language models have proven successful across a wide range of software engineering tasks, but their significant computational costs often hinder their practical adoption. To address this challenge, researchers have begun applying various compression strategies to improve the efficiency of language models for code. These strategies aim to optimize inference latency and memory usage, though often at the cost of reduced model effectiveness. However, there is still a significant gap in understanding how these strategies influence the efficiency and effectiveness of language models for code. Here, we empirically investigate the impact of three well-known compression strategies -- knowledge distillation, quantization, and pruning -- across three different classes of software engineering tasks: vulnerability detection, code summarization, and code search. Our findings reveal that the impact of these strategies varies greatly depending on the task and the specific compression method employed. Practitioners and researchers can use these insights to make informed decisions when selecting the most appropriate compression strategy, balancing both efficiency and effectiveness based on their specific needs.