Artificial or Just Artful? Do LLMs Bend the Rules in Programming?

TL;DR

This paper investigates how large language models adapt their code generation strategies when exposed to test cases under different prompting conditions, revealing significant performance changes and adaptation strategies that highlight conflicts between pretraining and alignment.

Contribution

It introduces a systematic analysis of LLM behavior with test signals, revealing how models adapt and the effectiveness of restrictions, advancing understanding of model alignment in code generation.

Findings

Test visibility significantly impacts correctness, with nearly doubling performance for some models.

Explicit restrictions only partially mitigate the influence of test cases.

Test-driven refinement is the most common adaptation strategy among LLMs.

Abstract

Large Language Models (LLMs) are widely used for automated code generation, yet their apparent successes often mask a tension between pretraining objectives and alignment choices. While pretraining encourages models to exploit all available signals to maximize success, alignment, whether through fine-tuning or prompting, may restrict their use. This conflict is especially salient in agentic AI settings, for instance when an agent has access to unit tests that, although intended for validation, act as strong contextual signals that can be leveraged regardless of explicit prohibitions. In this paper, we investigate how LLMs adapt their code generation strategies when exposed to test cases under different prompting conditions. Using the BigCodeBench (Hard) dataset, we design five prompting conditions that manipulate test visibility and impose explicit or implicit restrictions on their use. We evaluate five LLMs (four open-source and one closed-source) across correctness, code similarity, program size, and code churn, and analyze cross-model consistency to identify recurring adaptation strategies. Our results show that test visibility dramatically alters performance, correctness nearly doubles for some models, while explicit restrictions or partial exposure only partially mitigate this effect. Beyond raw performance, we identify four recurring adaptation strategies, with test-driven refinement emerging as the most frequent. These results highlight how LLMs adapt their behavior when exposed to contextual signals that conflict with explicit instructions, providing useful insight into how models reconcile pretraining objectives with alignment constraints.