An Empirical Study on the Effects of System Prompts in Instruction-Tuned Models for Code Generation

TL;DR

This study systematically evaluates how system prompts influence the performance of instruction-tuned models in code generation, revealing that prompt effectiveness varies with configuration, model size, language, and strategy, challenging some conventional assumptions.

Contribution

It provides a comprehensive empirical analysis of system prompt effects on code generation models, highlighting nuanced interactions and the importance of prompt engineering.

Findings

Increasing prompt constraint specificity does not always improve correctness.

Few-shot examples can degrade performance in larger models.

Java code generation is more sensitive to prompt variations than Python.

Abstract

Instruction-tuned Language Models (ILMs) have become essential components of modern AI systems, demonstrating exceptional versatility across natural language and reasoning tasks. Among their most impactful applications is code generation, where ILMs -- commonly referred to as Code Language Models (CLMs) -- translate human intent into executable programs. While progress has been driven by advances in scaling and training methodologies, one critical aspect remains underexplored: the impact of system prompts on both general-purpose ILMs and specialized CLMs for code generation. We systematically evaluate how system prompts of varying instructional detail, along with model scale, prompting strategy, and programming language, affect code assistant. Our experimental setting spans 360 configurations across four models, five system prompts, three prompting strategies, two languages, and two temperature settings. We find that (1) increasing system-prompt constraint specificity does not monotonically improve correctness -- prompt effectiveness is configuration-dependent and can help or hinder based on alignment with task requirements and decoding context; (2) for larger code-specialized models, few-shot examples can degrade performance relative to zero-shot generation, contrary to conventional wisdom; and (3) programming language matters, with Java exhibiting significantly greater sensitivity to system prompt variations than Python, suggesting language-specific prompt engineering strategies may be necessary.