The Rise of Language Models in Mining Software Repositories: A Survey

TL;DR

This survey reviews the adoption and impact of language models in mining software repositories, highlighting trends, challenges, and future research directions in the field.

Contribution

It provides a comprehensive taxonomy and analysis of 85 papers on LM applications in MSR, identifying key trends and open challenges.

Findings

LM adoption in MSR has grown rapidly since Transformer models emerged.

The survey categorizes LM applications and analyzes their evolution over time.

Open challenges include reproducibility and handling heterogeneous data.

Abstract

The Mining Software Repositories (MSR) field focuses on analysing the rich data contained in software repositories to derive actionable insights into software processes and products. Mining repositories at scale requires techniques capable of handling large volumes of heterogeneous data, a challenge for which language models (LMs) are increasingly well-suited. Since the advent of Transformer-based architectures, LMs have been rapidly adopted across a wide range of MSR tasks. This article presents a comprehensive survey of the use of LMs in MSR, based on an analysis of 85 papers. We examine how LMs are applied, the types of artefacts analysed, which models are used, how their adoption has evolved over time, and the extent to which studies support reproducibility and reuse. Building on this analysis, we propose a taxonomy of LM applications in MSR, identify key trends shaping the field, and highlight open challenges alongside actionable directions for future research.