Compressed code: the hidden effects of quantization and distillation on programming tokens

TL;DR

This paper systematically analyzes how various model compression techniques like quantization and distillation impact token representations and code generation quality in large language models, providing new insights and practical guidelines.

Contribution

It introduces a novel cold-start probability analysis method and evaluates the effects of multiple optimization techniques on token behavior and code generation.

Findings

Quantization and distillation significantly alter token distributions.

Model scaling and fine-tuning influence keyword coverage and token representations.

Guidelines are provided for maintaining code quality in compressed models.

Abstract

Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments.