GraphRepo: Fast Exploration in Software Repository Mining

TL;DR

GraphRepo is a unified, high-performance tool built on Neo4j that simplifies software repository data extraction, storage, and sharing, enabling scalable and flexible repository mining with easy integration into Python and big data ecosystems.

Contribution

It introduces GraphRepo, a modular, extensible platform that unifies repository mining processes using a graph database, improving performance and interoperability over project-specific solutions.

Findings

Fast querying of repository data demonstrated in benchmarks

Supports scalable exploration of large datasets

Enables easy data sharing and distribution

Abstract

Mining and storage of data from software repositories is typically done on a per-project basis, where each project uses a unique combination of data schema, extraction tools, and (intermediate) storage infrastructure. We introduce GraphRepo, a tool that enables a unified approach to extract data from Git repositories, store it, and share it across repository mining projects. GraphRepo usesNeo4j, an ACID-compliant graph database management system, and allows modular plug-in of components for repository extraction (drillers), analysis (miners), and export (mappers). The graph enables a natural way to query the data by removing the need for data normalisation. GraphRepo is built in Python and offers multiple ways to interface with the rich Python ecosystem and with big data solutions. The schema of the graph database is generic and extensible. Using GraphRepo for software repository mining offers several advantages versus creating project-specific infrastructure: (i) high performance for short-iteration exploration and scalability to large data sets (ii) easy distribution of extracted data(e.g., for replication) or sharing of extracted data among projects, and (iii) extensibility and interoperability. A set of benchmarks on four open source projects demonstrate that GraphRepo allows very fast querying of repository data, once extracted and indexed. More information can be found in the project's documentation (available at https://tinyurl.com/grepodoc) and in the project's repository (available at https://tinyurl.com/grrepo). A video demonstration isalso available online (https://tinyurl.com/grrepov)