Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation

TL;DR

This paper systematically investigates how quantization impacts the robustness of large language models in code generation, revealing that quantized models often outperform full-precision ones in adversarial and noise resilience.

Contribution

First comprehensive study on the robustness effects of quantization on LLMs in code generation, challenging assumptions about potential robustness degradation.

Findings

Quantized LLMs show higher adversarial robustness (51.59%) compared to full-precision models (42.86%).

Quantization generally improves LLMs' resilience to weight perturbations.

Results suggest quantization can enhance both efficiency and robustness of LLMs.

Abstract

Quantization has emerged as a mainstream method for compressing Large Language Models (LLMs), reducing memory requirements and accelerating inference without architectural modifications. While existing research primarily focuses on evaluating the effectiveness of quantized LLMs compared to their original counterparts, the impact on robustness remains largely unexplored.In this paper, we present the first systematic investigation of how quantization affects the robustness of LLMs in code generation tasks. Through extensive experiments across four prominent LLM families (LLaMA, DeepSeek, CodeGen, and StarCoder) with parameter scales ranging from 350M to 33B, we evaluate robustness from dual perspectives: adversarial attacks on input prompts and noise perturbations on model architecture. Our findings challenge conventional wisdom by demonstrating that quantized LLMs often exhibit superior robustness compared to their full-precision counterparts, with 51.59% versus 42.86% of our adversarial experiments showing better resilience in quantized LLMs. Similarly, our noise perturbation experiments also confirm that LLMs after quantitation generally withstand higher levels of weight disturbances. These results suggest that quantization not only reduces computational requirements but can actually enhance LLMs' reliability in code generation tasks, providing valuable insights for developing more robust and efficient LLM deployment strategies.