Context Engineering for AI Agents in Open-Source Software

TL;DR

This study investigates how open-source software projects adopt and structure AI context files like AGENTS$.$md, revealing significant variation and evolution over time, which impacts the effectiveness of AI agents in software development.

Contribution

The paper provides the first empirical analysis of AI context file adoption and structure in open-source projects, highlighting variability and evolution in content and presentation.

Findings

No standardized content structure in AI context files

High variation in how context is provided (descriptive, prescriptive, etc.)

Insights into how context files evolve over time

Abstract

GenAI-based coding assistants have disrupted software development. The next generation of these tools is agent-based, operating with more autonomy and potentially without human oversight. Like human developers, AI agents require contextual information to develop solutions that are in line with the standards, policies, and workflows of the software projects they operate in. Vendors of popular agentic tools (e.g., Claude Code) recommend maintaining version-controlled Markdown files that describe aspects such as the project structure, code style, or building and testing. The content of these files is then automatically added to each prompt. Recently, AGENTS$.$md has emerged as a potential standard that consolidates existing tool-specific formats. However, little is known about whether and how developers adopt this format. Therefore, in this paper, we present the results of a preliminary study investigating the adoption of AI context files in 466 open-source software projects. We analyze the information that developers provide in AGENTS$.$md files, how they present that information, and how the files evolve over time. Our findings indicate that there is no established content structure yet and that there is a lot of variation in terms of how context is provided (descriptive, prescriptive, prohibitive, explanatory, conditional). Our commit-level analysis provides first insights into the evolution of the provided context. AI context files provide a unique opportunity to study real-world context engineering. In particular, we see great potential in studying which structural or presentational modifications can positively affect the quality of the generated content.