Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

TL;DR

This study evaluates whether repository-level context files like AGENTS.md improve coding agent performance, finding that they often decrease success rates and increase costs, suggesting minimal, well-crafted context is preferable.

Contribution

The paper provides a rigorous evaluation of the effectiveness of repository context files for coding agents, revealing they can hinder performance and increase costs, contrary to common assumptions.

Findings

Context files often reduce task success rates.

Using context files increases inference costs by over 20%.

Both generated and human-written context files promote broader exploration.

Abstract

A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md, by either manually or automatically generating them. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files following agent-developer recommendations, and a novel collection of issues from repositories containing developer-committed context files. Across multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%. Behaviorally, both LLM-generated and developer-provided context files encourage broader exploration (e.g., more thorough testing and file traversal), and coding agents tend to respect their instructions. Ultimately, we conclude that unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements.