Repository Structure-Aware Training Makes SLMs Better Issue Resolver

TL;DR

This paper introduces ReSAT, a training method that enhances small language models' ability to resolve issues and understand repository structure by using specialized training data, narrowing the performance gap with larger models.

Contribution

ReSAT is a novel training approach that improves SLMs' performance on complex software development tasks by incorporating repository-aware data and multi-level localization techniques.

Findings

ReSAT significantly improves SLMs' issue resolution accuracy.

Enhanced long-context understanding in SLMs demonstrated on SWE-Bench and RepoQA.

ReSAT narrows the performance gap between SLMs and LLMs in repository-level tasks.

Abstract

Language models have been applied to various software development tasks, but the performance varies according to the scale of the models. Large Language Models (LLMs) outperform Small Language Models (SLMs) in complex tasks like repository-level issue resolving, but raise concerns about privacy and cost. In contrast, SLMs are more accessible but under-perform in complex tasks. In this paper, we introduce ReSAT (Repository Structure-Aware Training), construct training data based on a large number of issues and corresponding pull requests from open-source communities to enhance the model's understanding of repository structure and issue resolving ability. We construct two types of training data: (1) localization training data, a multi-level progressive localization data to improve code understanding and localization capability; (2) code edit training data, which improves context-based code editing capability. The evaluation results on SWE-Bench-verified and RepoQA demonstrate that ReSAT effectively enhances SLMs' issue-resolving and repository-level long-context understanding capabilities.