Do Agent Rules Shape or Distort? Guardrails Beat Guidance in Coding Agents

TL;DR

This study empirically evaluates how developer-provided rules influence AI coding agents, revealing that negative constraints improve performance while positive directives may hinder it, emphasizing the importance of rule type over quantity.

Contribution

First large-scale empirical analysis of developer rules for coding agents, showing negative constraints are beneficial and positive directives can be harmful, guiding safer agent configuration.

Findings

Rules improve performance by 7-14 percentage points.

Random rules are as effective as expert-curated ones.

Negative constraints are the only beneficial rule type.

Abstract

Developers increasingly guide AI coding agents through natural language instruction files (e.g., CLAUDE.md, .cursorrules), yet no controlled study has measured whether these rules actually improve agent performance or which properties make a rule beneficial. We scrape 679 such files (25,532 rules) from GitHub and conduct the first large-scale empirical evaluation, running over 5,000 agent runs with a state-of-the-art coding agent on SWE-bench Verified. Rules improve performance by 7--14 percentage points, but random rules help as much as expert-curated ones -- suggesting rules work through context priming rather than specific instruction. Negative constraints ("do not refactor unrelated code") are the only individually beneficial rule type, while positive directives ("follow code style") actively hurt -- a pattern we analyze through the lens of potential-based reward shaping (PBRS). Moreover, individual rules are mostly harmful in isolation yet collectively helpful, with no degradation up to 50 rules. These findings expose a hidden reliability risk -- well-intentioned rules routinely degrade agent performance -- and provide a clear principle for safe agent configuration: constrain what agents must not do, rather than prescribing what they should.