Does the First Response Matter for Future Contributions? A Study of First Contributions

TL;DR

This study analyzes how first responses to initial contributions in open source projects influence future participation, highlighting the emotional impact of responses and the predictive challenges of future engagement.

Contribution

It provides a large-scale analysis of first responses and their emotional content, and evaluates machine learning models for predicting future contributions based on initial interactions.

Findings

Most first responses are positive but less responsive.

Negative responses often aim to evoke constructive or critical emotions.

Predictive models for future contributions have moderate accuracy (F1 score 0.6171).

Abstract

Open Source Software (OSS) projects rely on a continuous stream of new contributors for their livelihood. Recent studies reported that new contributors experience many barriers in their first contribution, with the social barrier being critical. Although a number of studies investigated the social barriers to new contributors, we hypothesize that negative first responses may cause an unpleasant feeling, and subsequently lead to the discontinuity of any future contribution. We execute protocols of a registered report to analyze 2,765,917 first contributions as Pull Requests (PRs) with 642,841 first responses. We characterize most first response as being positive, but less responsive, and exhibiting sentiments of fear, joy and love. Results also indicate that negative first responses have the literal intention to arouse emotions of being either constructive (50.71%) or criticizing (37.68%) in nature. Running different machine learning models, we find that predicting future interactions is low (F1 score of 0.6171), but relatively better than baselines. Furthermore, an analysis of these models show that interactions are positively correlated with a future contribution, with other dimensions (i.e., project, contributor, contribution) having a large effect.