Ethical Classification of Non-Coding Contributions in Open-Source Projects via Large Language Models

TL;DR

This paper presents a method using Large Language Models to classify the ethical quality of non-coding contributions in open-source projects, aiming to support community standards and conduct enforcement.

Contribution

It introduces a novel LLM-based classification approach guided by ethical metrics derived from the Contributor Covenant for OSS contributions.

Findings

Effective classification of ethical contributions achieved

Guided prompt engineering improves model accuracy

Supports enforcement of codes of conduct in OSS

Abstract

The development of Open-Source Software (OSS) is not only a technical challenge, but also a social one due to the diverse mixture of contributors. To this aim, social-coding platforms, such as GitHub, provide the infrastructure needed to host and develop the code, but also the support for enabling the community's collaboration, which is driven by non-coding contributions, such as issues (i.e., change proposals or bug reports) or comments to existing contributions. As with any other social endeavor, this development process faces ethical challenges, which may put at risk the project's sustainability. To foster a productive and positive environment, OSS projects are increasingly deploying codes of conduct, which define rules to ensure a respectful and inclusive participatory environment, with the Contributor Covenant being the main model to follow. However, monitoring and enforcing these codes of conduct is a challenging task, due to the limitations of current approaches. In this paper, we propose an approach to classify the ethical quality of non-coding contributions in OSS projects by relying on Large Language Models (LLM), a promising technology for text classification tasks. We defined a set of ethical metrics based on the Contributor Covenant and developed a classification approach to assess ethical behavior in OSS non-coding contributions, using prompt engineering to guide the model's output.