On The Importance of Reasoning for Context Retrieval in Repository-Level Code Editing

TL;DR

This paper investigates the role of reasoning in context retrieval for repository-level code editing, highlighting its benefits and limitations in improving context precision and sufficiency.

Contribution

It decouples context retrieval from other components, providing insights into reasoning's impact and outlining the role of specialized tools in codebase navigation.

Findings

Reasoning improves context precision.

Current methods lack ability to determine context sufficiency.

Specialized tools are crucial for effective context gathering.

Abstract

Recent advancements in code-fluent Large Language Models (LLMs) enabled the research on repository-level code editing. In such tasks, the model navigates and modifies the entire codebase of a project according to request. Hence, such tasks require efficient context retrieval, i.e., navigating vast codebases to gather relevant context. Despite the recognized importance of context retrieval, existing studies tend to approach repository-level coding tasks in an end-to-end manner, rendering the impact of individual components within these complicated systems unclear. In this work, we decouple the task of context retrieval from the other components of the repository-level code editing pipelines. We lay the groundwork to define the strengths and weaknesses of this component and the role that reasoning plays in it by conducting experiments that focus solely on context retrieval. We conclude that while the reasoning helps to improve the precision of the gathered context, it still lacks the ability to identify its sufficiency. We also outline the ultimate role of the specialized tools in the process of context gathering. The code supplementing this paper is available at https://github.com/JetBrains-Research/ai-agents-code-editing.