Processing Large Datasets of Fined Grained Source Code Changes

TL;DR

This paper introduces extensions to the CodeDistillery framework that enable efficient processing and manipulation of large datasets of fine-grained source code changes, facilitating scalable repository mining in Big Code research.

Contribution

The paper presents new data manipulation capabilities integrated into CodeDistillery, allowing automated processing of millions of source code change records.

Findings

Successfully processed dozens of millions of fine-grained source code changes.

Enhanced automation streamlines repository mining workflows.

Framework extensions support large-scale Big Code studies.

Abstract

In the era of Big Code, when researchers seek to study an increasingly large number of repositories to support their findings, the data processing stage may require manipulating millions and more of records. In this work we focus on studies involving fine-grained AST level source code changes. We present how we extended the CodeDistillery source code mining framework with data manipulation capabilities, aimed to alleviate the processing of large datasets of fine grained source code changes. The capabilities we have introduced allow researchers to highly automate their repository mining process and streamline the data acquisition and processing phases. These capabilities have been successfully used to conduct a number of studies, in the course of which dozens of millions of fine-grained source code changes have been processed.