Automated Extraction and Analysis of Developer's Rationale in Open Source Software

TL;DR

This paper presents an automated method using advanced language models to extract and analyze developers' rationale in open source projects, helping identify conflicts and potential design erosion over time.

Contribution

It introduces a novel automated rationale extraction approach based on Kantara architecture, leveraging pre-trained and large language models for conflict detection in open source software.

Findings

Feasible extraction of rationale sentences demonstrated on Linux Kernel's OOM-Killer module.

Effective detection of reasoning conflicts and potential design erosion.

Approach generalizes well to multiple active open source projects.

Abstract

Contributors to open source software must deeply understand a project's history to make coherent decisions which do not conflict with past reasoning. However, inspecting all related changes to a proposed contribution requires intensive manual effort, and previous research has not yet produced an automated mechanism to expose and analyze these conflicts. In this article, we propose such an automated approach for rationale analyses, based on an instantiation of Kantara, an existing high-level rationale extraction and management architecture. Our implementation leverages pre-trained models and Large Language Models, and includes structure-based mechanisms to detect reasoning conflicts and problems which could cause design erosion in a project over time. We show the feasibility of our extraction and analysis approach using the OOM-Killer module of the Linux Kernel project, and investigate the approach's generalization to five other highly active open source projects. The results confirm that our automated approach can support rationale analyses with reasonable performance, by finding interesting relationships and to detect potential conflicts and reasoning problems. We also show the effectiveness of the automated extraction of decision and rationale sentences and the prospects for generalizing this to other open source projects. This automated approach could therefore be used by open source software developers to proactively address hidden issues and to ensure that new changes do not conflict with past decisions.