Communication and Code Dependency Effects on Software Code Quality: An Empirical Analysis of Herbsleb Hypothesis

TL;DR

This study empirically investigates the impact of developer communication and code dependency patterns on software quality, finding that communication quality is more strongly linked to fewer bugs than code dependency centrality.

Contribution

It introduces two metrics for social interaction and hero-ness, and demonstrates that communication patterns are more predictive of code quality than hero-centric code changes.

Findings

Social-ness correlates more strongly with fewer bugs than hero-ness.

Effective communication flows reduce bug counts in open source projects.

Code dependency centrality is less indicative of code quality than social interactions.

Abstract

Prior literature has suggested that in many projects 80\% or more of the contributions are made by a small called group of around 20% of the development team. Most prior studies deprecate a reliance on such a small inner group of "heroes", arguing that it causes bottlenecks in development and communication. Despite this, such projects are very common in open source projects. So what exactly is the impact of "heroes" in code quality? Herbsleb argues that if code is strongly connected yet their developers are not, then that code will be buggy. To test the Hersleb hypothesis, we develop and apply two metrics of (a) "social-ness'"and (b) "hero-ness" that measure (a) how much one developer comments on the issues of another; and (b) how much one developer changes another developer's code (and "heroes" are those that change the most code, all around the system). In a result endorsing the Hersleb hypothesis, in over 1000 open source projects, we find that "social-ness" is a statistically stronger indicate for code quality (number of bugs) than "hero-ness". Hence we say that debates over the merits of "hero-ness" is subtly misguided. Our results suggest that the real benefits of these so-called "heroes" is not so much the code they generate but the pattern of communication required when the interaction between a large community of programmers passes through a small group of centralized developers. To say that another way, to build better code, build better communication flows between core developers and the rest. In order to allow other researchers to confirm/improve/refute our results, all our scripts and data are available, on-line at https://github.com/Anonymous633671/A-Comparison-on-Communication-and-Code-Dependency-Effects-on-Software-Code-Quality.

Tables

N	Number of Developers
P	80, 85, 90, and 95 Percentile
Rank()	The percentile rank of a score is the percentage of scores in its frequency distribution that is equal to or lower than it.
a	Adjacency matrix for the graph where $a_{ij}>0$ denotes a connection

Table 2. Table 1: The table summarizes of Fig  6 , Fig  7 and stratifies the data according to 25th, 50th, and 75th percentile of the developers.

		Percentile
Metric	Group	25th	50th	75th
	Hero	0.52	0.58	0.53
	Non-Hero	0.67	0.75	1.0
Code Interaction	Ratio	1.3	1.3	1.9
	Hero	0.44	0.5	0.5
	Non-Hero	0.67	0.75	0.67
Social Interaction	Ratio	1.5	1.5	1.3

Table 3. Table 2: This figure shows the result of statistical significance test and an effect size test on 9 different groups used to study the Herbsleb hypothesis. In this figure the “group” column represents the 9 different groups in this research question, where the first character represents the code node degree, while the later is social node degree.

rank	group (code,social)	median	IQR
1	L,H	30	29
1	M,H	38	28
2	L,M	38	31
2	H,H	42	18
2	M,M	46	21
2	H,M	46	21
2	H,L	48	30
3	M,L	52	29
4	L,L	67	45

Equations

$$\text{mbox}NodeDegreeof{N}_i=D(N_i)=\sum _{j=1}^n{a}_{ij}$$

$$\text{mbox}Hero=Rank\left(D(N_i)\right)>\frac{P}{100}\ast (N+1)$$

$$\text{mbox}Non-Hero=Rank\left(D(N_i)\right)<\frac{P}{100}\ast (N+1)$$

Full text

Herbsleb argues that if code is strongly connected yet their developers are not, then that code will be buggy. To test the Hersleb hypothesis, we develop and apply two metrics of (a) “social-ness’ and (b) “hero-ness” that measure (a) how much one developer comments on the issues of another; and (b) how much one developer changes another developer’s code (and “heroes” are those that change the most code, all around the system). In a result endorsing the Hersleb hypothesis, in over 1000 open source projects, we find that “social-ness” is a statistically stronger indicate for code quality (number of bugs) than “hero-ness”.

Hence we say that debates over the merits of “hero-ness” is subtly misguided. Our results suggest that the real benefits of these so-called “heroes” is not so much the code they generate but the pattern of communication required when the interaction between a large community of programmers passes through a small group of centralized developers. To say that another way, to build better code, build better communication flows between core developers and the rest.

In order to allow other researchers to confirm/improve/refute our results, all our scripts and data are available, on-line at our online Github repository.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

††journal: Journal of Systems and Software

1 Introduction

Developers talk amongst themselves to coordinate development activities such as prioritize and assign tasks, comprehend requirements better, discuss candidate solutions to complex problems, and like. Healthy communication is likely to result in good quality software. Sound team communication is vital to the success of any software project involving multiple developers. James Herbsleb [1], in his ICSE’14 keynote said:

If two code sections communicate but the programmers of those two sections do not, then that code section is more likely to be buggy.

Many prior literature shows in many projects 80% or more of the contributions are made by only 20% of the developers. These developers are often referred to ”hero” developers and the projects as “hero” projects. In the literature, hero projects are deprecated since it is said, they are like bottlenecks that slow down the project development process and causes information loss [2, 3, 4, 5, 6, 7]. From the perspective of Herbsleb hypothesis, “hereos” are a problem if they result in less communication between developers.

Recent studies have motivated a re-examination of the implications of heroes. In 2018, Agrawal et al. [8] studied 661 open source projects and 171 in-house proprietary projects. In that sample, over 89% of projects were hero-based111This text use “hero” for women and men since recent publications use it to denote admired people of all genders– see bit.ly/2UhJCek.. Since the widespread prevalence of heroes is at odds with established wisdom in the SE literature. The usual stance is to warn against heroes since they may become bottlenecks in development and communication [2, 3, 4, 5, 6, 9, 10, 11]. Hence in software engineering research, it is a pressing issue to understand why so many projects are hero-based and what are implication of communication between these developers have on code quality (introduction of bugs in the code base). In most projects only a small group of developers are making the most contributions (writing the codes), then what happens to the code quality when they do and do not communicate with each other properly? Also, what about the code quality of developers who do not contribute much?

To that end, this paper tests the “Herbsleb Hypothesis”:

• •

We collect data on over 1000 open source Github projects.

• •

We show that a majority of our projects are hero projects (i.e., majority percentage of contributions are made by only a small percentage of developers)

• •

We explain why heroes are important. As shown below, when developer’s codes interact with each other, if there is significant social interaction (social communication) between them as well they tends to introduce much less bugs into the code base than when there is not.

As part of this research, we ask three research questions -

RQ1: How common are projects with highly centralized social and code communication structure (i.e., hero projects)? Here we see a overwhelming presence of projects with highly centralized social and code communication structure in open source software community. That is, usually, there are very few people in each project responsible for most of the work.

RQ2: What impact does high social and code communication have on code quality? Reflecting on who writes most of the code is just as insightful as reflecting on who participates in most of the discussion about the code.

RQ3: Do the results support Herbsleb Hypothesis? Our conclusion will be “yes”.

The rest of the paper is structured as follows. Section 2 provides background information that directly relates to our research questions, in addition to laying out the motivation behind our work. Section 3.1 explains the data collection process and in Section 3.2, a detailed description of our experimental setup and data is given, along with our performance criteria for evaluation is presented. It is followed by Section 4 the results of the experiments and answers to some research questions. threats to validity. Threats to validity are discussed in Section 5, which is followed by our conclusions, in Section 6.

1.1 Relation to Prior Work

This paper extends prior work by Agrawal et al. [8]. That prior worked only looked at half the data explored here. Also, that work failed to detect the importance of the social effects reported here (so they could show empirically that hero projects existed but they did not recognize the important underlying social effects).

1.2 Definitions

Before beginning, we make some definitional points. When we say 1000+ projects, that is shorthand for the following. Our results used the intersection of two graphs of code interaction graph (of who writes what and whose code) with social interaction graph (who discusses what issue and with whom) from 1037 projects. Secondly, by code interaction graphs and social interaction graphs, we mean the following. Each graph has nodes and edges $\{N,E\}$. For code interaction graphs:

• •

Individual developers have their own node $N_{c}$;

• •

The edge $E_{c}$ connects two nodes and indicates if ever one developer has changed another developer’s code. $W_{c}$ denotes how much one developer has changed another developer’s code.

For social interaction graphs like Figure 1:

• •

A node $N_{s}$ is created for each individual who has created or commented on an issue.

• •

An edge $E_{s}$ indicates communication between two individuals (as recorded in the issue tracking system.) If this happens $N$ times then the weight $W_{s}=N$.

Thirdly, we have defined heroes based on code contribution and communication. From the “Code interaction graph”, the developers who contribute more than 80% are hero contributors, and in the “Social interaction graph”, the developers who are making 80% of the communication are hero communicators. Both are “hero developers” for us. As we show in §4, both these definitions of heroes are insightful since more can be predicted about a project using both definitions that it either is applied separately.

2 Background And Prior Work

2.1 Herbsleb Hypothesis (and Analogs)

At his ICSE’14 keynote, James Hersleb defined coding to be a socio-technical process where code and humans interact. According to the Hersleb hypothesis [1], the following anti-pattern is a strong predictor for defects:

• •

If two code sections communicate…

• •

But the programmers of those two sections do not…

• •

Then that code section is more likely to be buggy.

To say that another way:

*Coding is a social process and better code arises from better social interactions. *

Communication plays an important role in software development projects and the quality of communication has been found as determinant of project success [12, 13, 14]. The dynamic nature of open source software development makes the communication process between developers volatile [15], consequently affecting a teams ability to effectively communicate and coordinate. These difficulties are more prominent in platforms like Github where we have distributed teams as developers work in geographically remote environments. The importance of communication in successful project completion is also well documented [16, 17, 18]. Many other researchers offer conclusions analogous to the Herbsleb hypothesis. Developer communication/interaction is often cited as one of the most important factors for a successful software development [19, 20, 21, 22]. Many researchers have shown that successful communication between developers and adequate knowledge about the system plays a key role in successful software development [23, 24, 25]. As reported as early as 1975 in Brooks et al. text “The Mythical Man Month” [26], communication failure can lead to coordination problems, lack of system knowledge in the projects as discussed by Brooks et al. in the Mythical Man-Month.

The usual response to the above argument is to improve communication by “smoothing it out”, i.e. by deprecating heroes since, it is argued, that encourages more communication across an entire project [2, 3, 4, 5, 6]. The premise of “smoothing it out” is that heroes are bad and should be deprecated. This paper tries to verify whether or not this premise holds true for open source GitHub projects or not.

2.2 Heroism in Software Development

Heroism in software development is a widely studied topic. Various researchers have found the presence of heroes in software projects. For example, In 1975 Brooks [27] proposed basing programming teams around a small number of “chief programmers” (which we would call “heroes”) who are supported by a large number of support staff (Brooks’s analogy was the operating theater where one surgeon is supported by one or two anesthetists, several nurses, clerical staff, etc). Also, the Agile Alliance [28] and Bach et al. [29] believed that heroes are the core ingredients in successful software projects saying “… the central issue is the human processor - the hero who steps up and solves the problems that lie between a need expressed and a need fulfilled.” In 2002, Mockus et al. [30] analyzed Apache and Mozilla projects to show the presence of heroes in those projects and reported, their positive influence on projects. Also in 2002, Koch et al. [31] studied the GNOME project and showed the presence of heroes throughout the project history. They conjectured (without proof) that the small number of hero developers may allow easy communication and collaboration. Interestingly, they also showed there is no relation between a developer’s time in the project and being a hero developer. In 2005, Krishnamurthy [32] studied 100 open source projects to find that a few individuals are responsible for the main contribution of the project in most of the cases. In 2006 and 2009, Robles et al. [33, 10] explored in their research the presence and evolution of heroes in open source software community. In 2013, Peterson analyzed the software development process on GitHub and found out a pattern that most development is done by a small group of developers [34]. He stated that for most of the GitHub projects, 95-100% of commits come from very few developers. Also, as mentioned in the introduction, In 2018, Agarwal et al. [8] stated that hero projects are very common. As software projects grow in size, nearly all projects become hero projects.

Most prior researchers deprecate heroism in software projects arguing that:

• •

Having most of the work is dependent on a small number of heroes can become a bottleneck that slows down project development  [2, 5, 4, 3, 6].

• •

In the case of hero projects, there is less collaboration between team members since there are few active team members. So, it is argued, heroes are negatively affecting the collaboration which is essential to the process of software development (and, more importantly, open source software development)[35, 36].

This second point is problematic since, in the literature, studies that analyze distributed software development on social coding platforms like GitHub and Bitbucket [37, 38] remark on how social collaborations can reduce the cost and efforts of software development without degrading the quality of software. Distributed coding effort is beneficial for agile community-based programming practices which can in turn have higher customer satisfaction, lower defect rates, and faster development times  [39, 40]. Customer satisfaction, it is argued, is increased when faster development leads to:

• •

Increasing the number of issues/bugs/enhancements being resolved [30, 41, 42, 43, 44, 45].

• •

Lowering the issues/bugs/enhancements resolution times [41, 14].

Even more specifically, as to issues related to heroes, Bier et al. warn when a project becomes complicated, it is always better to have a community of experts rather than having very few hero developers  [2]. Willams et al. have shown that hero programmers are often responsible for poorly documented software systems as they remain more involved in coding rather than writing code related documents  [4]. Also, Wood et al. [6] caution that heroes are often code-focused but software development needs workers acting as more than just coders (testers, documentation authors, user-experience analysts).

Our summary of the above is as follows: with only isolated exceptions, most of the literature deprecates heroes. Yet as discussed in the introduction, many studies indicate that heroic projects are quite common. This mismatch between established theory and a widely observed empirical effect prompted a re-examination of previous beliefs and the analysis discussed in this paper.

3 Methodology

Now to understand how communication between developers affects code quality, we need to perform different steps to -

• •

Collect data about which developers coded which part of a project.

• •

Collect data about how each developer’s code interacted with other developers.

• •

Collect data about how developers communicated with each other socially.

• •

Identify developers who contributes more and who contributes less both in-terms of code and communication.

• •

Collect and identify presence of bugs in a code base and identify who is responsible for the bug.

The following section describes how we performed each steps in details.

3.1 Data Collection

Figure 2 summarizes the Github data we used for this study. To understand this figure, we offer the following definitions:

• •

Release: (based on Git tags) marks a specific point in the repository’s history. The number of releases defines different versions published, which signifies a considerable amount of changes done between each version.

• •

Duration: length of a project from its inception to the current date or project archive date. It signifies how long a project has been running and in the active development phase.

• •

Stars: signifies the number of people liking a project or use them as bookmarks so they can follow what’s going on with the project later.

• •

Forks: A fork is a copy of a repository. Forking a repository allows users to freely experiment with changes without affecting the original project. This signifies how people are interested in the repository and actively thinking of modification of the original version.

• •

Watcher: Watchers are GitHub users who have asked to be notified of activity in a repository, but have not become collaborators. This is a representative of people actively monitoring projects, because of possible interest or dependency.

• •

Developer: Developers are the contributors to a project, who work on some code, and submit the code using commits to the codebase. The number of developers signifies the interest of developers in actively participating in the project and volume of the work.

Figure 3 shows that the projects we chose for our experiment comprise different languages. Note that we did not use all the data in Github. GitHub has over 100 million repositories as of the time of this collection so we only use data from the “GitHub showcase project” list. Many of these projects contain very short development cycles; are used for personal use; and are not be related to software development. Such projects may bias research findings. To mitigate that, we filter out projects using the standard “sanity checks” recommended in the literature [46, 47]:

• •

Collaboration: refers to the number of pull requests. This is indicative of how many other peripheral developers work on this project. We required all projects to have at least one pull request.

• •

Commits: The project must contain more than 20 commits.

• •

Duration: The project must contain software development activity of at least 50 weeks.

• •

Issues: The project must contain more than 10 issues.

• •

Personal Purpose: The project must have at least eight contributors.

• •

Software Development: The project must be a placeholder for software development source code.

• •

Project Documentation Followed: The projects should follow proper documentation standards to log proper commit comment and issue events to allow commit issue linkage.

• •

Social network validation: The Social Network that is being built should have at least 8 connected nodes in both the communication and code interaction graph (this point is discussed further in 3.2.2 and 3.2.3).

For each of the selected project, the study recreates all the committed files to identify code changes in each commit file and identifies developers using the GitHub API, then downloads issue comments and events for a particular project, and uses the git log command to mine the git commits added to the project throughout the project lifetime. Using the information from each commit message, this study uses keyword based identifier [48, 49, 50] to label commits as buggy commit or not by identifying commits which were used to fix some bugs in the code and then identifies the last commit which introduced the bug. This commit is labeled as a buggy commit.

3.2 Information Extraction

Here we discuss the process of extracting the relevant features and information regarding communication and code quality from the Github information collected [51, 52, 53]

3.2.1 Code Interaction Extraction

Recall that the developer code interaction graph records who touched what and whose code, where a developer is defined as a person who has ever committed any code into the codebase. We create that graph as follows:

• •

Project commits were extracted from each branch in git history.

• •

Commits are extracted from the git log and stored in a file system.

• •

To access the file changes in each commit we recreate the files that were modified in each commit by (a) continuously moving the git head chronologically on each branch. Changes were then identified using git diff on two consecutive git commits.

• •

The graph is created by going through each commit and adding a node for the committer. Then we use git blame on the lines changed to find previous commits following a similar process to the SZZ algorithm [54]. We identify all the developers of the commits from git blame and add them as a node as well.

• •

After the nodes are created, directed edges were drawn between the developer who changed the code, to whose code was changed. Those edges were weighted by the change size between the developers.

3.2.2 Social Interaction Extraction

Recall that the developer social interaction graph records who talked to each other via issue comments. We create that graph as follows:

• •

A node is created for the person who has created the issue, then another set of nodes are created for each person who has commented on the issue. So essentially in the Social interaction graph, each node in the graph is any person (developer or non-developer) who has ever created an issue or commented on an issue.

• •

The nodes are connected by directed edges, which are created by (a) connecting from the person who has created the issue to all the persons who have commented on that issue and (b) creating edges from the person commenting on the issue to all other persons who have commented on the issue, including the person who has created the issue.

• •

The edges are weighted by the number of comments by that person.

• •

The weights are updated using the entire history of the projects, as per Figure 4.

3.2.3 Defect Data Extraction

This study explores the effects of social and code communication to assess code quality, by measuring the percentage of buggy commits introduced by developers (hero and non-hero developers), but to do so we do need to identify the commits that introduced the bug in the code from the historic project data. This is a challenging task since there is no direct way to find the commits or the person who is responsible for the bug/issue introduction. Hence, our scripts proceed as follows:

• •

Starting with all the commits from git log, we identify the commit messages.

• •

Next, to use the commit messages for labeling, we apply natural language processing [49, 48] (to do stemming, lemmatization, and lowercase conversion to normalize the commit messages).

• •

Then to identify commit messages which are representation of bug/issue fixing commits, a list of words and phrases is extracted from previous studies of 1000+ projects (Open Source and Enterprise). The system checked for these words and phrases in the commit messages and if found, it marks these as commits which fixed some bugs.

• •

To perform a sanity check, 5% of the commits were manually verified by 7 graduate students using random sampling from different projects. Disagreements between manual labeling and keyword based labeling were further verified and keywords were added or removed to improve performance.

• •

These labeled commits were then processed to extract the file changes as the process mentioned in section 3.2.1.

• •

Finally, git blame is used to go back in the git history to identify a responsible commit where each line was created or changed last time.

By this process, commits that were responsible for introduction of the bugs in the system/project can be found. We label these commits as “buggy commits” and label the author of the commit as the “person responsible” for introducing the bug.

3.2.4 Final Feature Extraction

To assess the prevalence of heroes in the software projects, we joined all the metrics shown above. Specifically, using both social and code interaction graph, we calculated the node degree (number of edges touching a vertex) of the graphs (and note that vertices with a higher degree represent more communication or interaction).

For the sake of completeness, we varied our threshold of “hero” to someone belonging to 80%, 85%, 90%, and 95% of the communication. In our studies, top contributors (or heroes) and non-heroes were defined as :

[TABLE]

[TABLE]

[TABLE]

where:

Using these data and by applying the hero definition from formula (2) and (3) (look at the top 20%, 15%, 10%, and 5%), we can find the developers who are responsible for 80%, 85%, 90%, and 95% of the work. We use this to categorize the developers into 2 groups:

• •

The hero developers; i.e. the core group of the developers of a certain project who make regular changes in the codebase. In this study, this is represented by the developers whose node degree is above 80, 85, 90, and 95th percentile of the node degree (developers’ communication and code interaction of the system graph).

• •

The non-hero developers are all other developers; i.e. developers associated with nodes with a degree below the respective threshold percentile.

Following this for each selected project, we merge the data collected in section 3.2.3 and section 3.2.1 to find each developer’s code contribution according to the code interaction graph. A similar process is followed for section 3.2.2 and section 3.2.1 in the social interaction graph. Using the above mentioned data, we can validate the code and social contribution of each developer along with their bug introduction percentage. This information will help us to answer the research questions asked in this study.

4 Results

Our results are structured around three research questions:

RQ1:

How common are projects with highly centralized social and code communication structure (i.e., hero projects)?

RQ2:

What impact does high social and code communication have on code quality?

RQ3:

Do the results support Herbsleb Hypothesis?

RQ1: How common are projects with highly centralized social and code communication structure (i.e., hero projects)?

Figure 5 shows the result for the number of projects where majority of social and code communication is done by only a few developers. We mark these projects as as hero and non-hero based on a varying threshold of $x$ percentage of work (both social and code communication) done by $y$ percentage of developers. Here we vary the $x$ percentage (that is the work done) from 80 to 95%, while we vary $y$ (i.e., percentage of developers) from 20 to 5%. The clear conclusion from this figure is that the phenomenon that we have defined as “hero“ is very common. In fact, the phenomenon may be more pronounced than previously reported. Even when we require heroes to be involved in 95% of the communication (which is a large amount), we find that majority of the projects studied here exhibits a very centralized social and code communication structure.

[TABLE]

RQ2: What impact does high social and code communication have on code quality?

RQ2 explores the effect of high social and code communication (heroism) have on code quality. In this study, we created the developer social interaction graph and developer code interaction graph, then identified the developer responsible for introducing those bugs into the codebase. Then we find the percentage of buggy commits introduced by those developers by checking (a) the number of buggy commits introduced by those developers and (b) their number of total commits.

Fig 6 and Fig 7 shows the comparison between the performance developers who had made high social and code communication (marked as hero) vs who had made little social and code communication (marked as non-hero) to the project. In those figures:

• •

The x-axis is different projects used in this study.

• •

The y-axis represents the median of the bug introduction percentage for all hero and non-hero developers for each project respectively.

Here projects are sorted by the number of non-hero developers. In those charts, we note that:

• •

There exists a large number of non-heroes who always produce buggy commits, 100% of the time (evidence: the flat right-hand-side regions of the non-hero plots in both figures). That population size of “always buggy” is around a third in Fig 6 and a fourth in Fig 7.

• •

To say the least, heroes nearly always have fewer buggy commits than non-heroes. The 25th, 50th, 75th percentiles for both groups are shown in Table 1. This table clearly shows why heroes are so prevalent, they generate commits that are dramatically less buggy than non-heroes, regardless of the size of the project.

The other thing to note from Fig 6 and Fig 7 is that they are nearly identical. That is, no matter how we define “hero”, we reach the same conclusions. Hence we say -

[TABLE]

RQ3: Do the results support Herbsleb Hypothesis?

In this research question, we explored the Herbsleb hypothesis [1] from section 2; i.e. does lack of communication between developers predict for bugs in the code? To do that for 1,037 projects, we discretized the developers into 3 groups (i.e. High, Medium, and Low) based on their code contribution (Code Node Degree) and social communication frequency (Social Node Degree). In Table 2, group HH (High, High) represents the developers who have high code contribution and social communication frequency and the value in the cell is median bug introduction percentage, while group LL (Low, Low) represents the developers who have low code contribution and low social communication frequency.

Table 2 shows the result of a statistical test performed on the 9 different groups representing different frequency and volume of communication. The “rank” column of that table shows a statistical analysis where one row has a higher rank than the previous one only if the two rows are

• •

Statistically significant different. For this test, we used the Scott-Knot method recommended by Angelis and Mittas at TSE’13 [55].

• •

And that difference is not a small effect. For this effect size test, we used the A12 test recommended by Angelis and Briand at ICSE’11 [56]

These statistical tests were selected since they were non-parametric; i.e. they do not assume normal distributions.

To summarize the effect reported in Table 2, we need to look at the difference between the highest and lowest ranks:

• •

In the groups with the highest defects and highest rank, the social and coding groups are both low. That is non-hero-ness (for both code and social interaction) is associated with the worst quality code.

• •

In the groups with the lowest defects and lowest rank, the social group is always high while the code groups are either low or medium. That is, extensive social interaction even with low to medium code interaction is associated with the best quality code.

From the above, we can say that:

• •

These results support the Herbsleb hypothesis, which suggests communication between developers is an important factor and lesser social communication can lead to more bugs.

• •

Also, prior definitions of “hero” based just on code interaction need now to be augmented. As shown in Table 2, social hero-ness is much more predictive for bugs than merely reflecting. on code hero-ness.

This finding leads to the following conjecture (to be explored in future work): the best way to reduce communication overhead and to decrease defects is to centralize the communicators. In our data, commits with lower defects come from the small number of hero developers who have learned how to talk to more people. Hence, we would encourage more research into better methods for rapid, high-volume, communication in a one-to-many setting (where the “one” is the hero and the “many” are everyone else). In summary, we can say

[TABLE]

5 Threats to Validity

5.1 Sampling Bias

Our conclusions are based on 1000+ open source GitHub projects that started this analysis. It is possible that different initial projects would have lead to different conclusions. That said, our initial sample is very large so we have some confidence that this sample represents an interesting range of projects.

5.2 Evaluation Bias

In RQ1, RQ2, and RQ3, we said that projects with highly centralized social and code communication structure very common, that is to say hero projects are common in open source community. Developers who are responsible for majority of the work (both social and code) are responsible for far less bug introduction than non-hero developers. It is possible that using other metrics222E.g. do heroes reduce productivity by becoming bottleneck? then there may well be a difference in these different kinds of projects. But measuring people resources only by how fast releases are done or issues are fixed may not be a good indicator of having heroes in a team. This is a matter that needs to be explored in future research.

Another evaluation bias as we report cumulative statistics of lift curves where other papers reported precision and recall. The research in this field is not mature enough yet for us to say that the best way to represent results is one way versus another. Here we decided to use lift curves since, if we used precision and recall, we had to repeat that analysis at multiple points of the lift curve. We find our current lift curves are a succinct way to represent our results.

5.3 Construct Validity

At various places in this report, we made engineering decisions about (e.g.) team size; and (e.g.) what constitutes a successful project. While those decisions were made using advice from the literature (e.g. [57]), we acknowledge that other constructs might lead to different conclusions.

That said, while we cannot prove that all of our constructs are in any sense “optimal”, the results of Table 2 suggest that our new definition of social hero-ness can be more informative than constructs used previously in the literature (that defined “hero” only in terms of code interaction).

Another issue about our construct validity is that we have relied on a natural language processor to analyze commit messages to mark them as buggy commits. These commit messages are created by the developers, and may or may not contain a proper indication of if they were used to fix some bugs. There is also a possibility that the team of that project might be using different syntax to enter in commit messages.

Yet another threat to construct validity is that we did not consider the different roles of the developers. We had trouble extracting that information from our data source, we found that people have multiple roles particularly our heroes who would often step in and assist in multiple activities. Nevertheless the exploration of different roles would be an interesting study.

5.4 External Validity

Previously Agrawal et al. were able to comment on the effects of heroes in open and closed source projects. That research group was fortunate enough to work on-site at a large open source software company. We were not as fortunate as them. We, therefore, acknowledge our findings (from open source projects) may not be the same for closed source projects.

Similarly, we have used GitHub issues and comments to create the communication graph, it is possible that the communication was not made using these online forums and was done with some other medium. To reduce the impact of this problem, we did take precautionary steps to (e.g.,) include various tag identifiers of bug fixing commits, did some spot check on projects regarding communication, etc.

Our conclusion shows that almost all (when experimenting with 80%, 85%, 90% threshold) of our sample projects are hero dominated. In case of large size public GitHub projects, there are official administrators and maintainers who are responsible for issue labeling or assigning. So, they frequently comment on all of the issues but though they are not active developers. These people should not be considered hero developers. Finding these people needs manual inspection which is not possible for 1000+ projects. We decide to put it as a limitation of our study as we deal with a huge number of projects.

We do not isolate hero projects and non-hero projects and look into them separately because there are very few non-hero projects and also there are a lot of developers who work on a large number of projects (some of them are hero projects and some of them are not).

6 Conclusion

In this study we investigated the impact of social and code communication on software code quality, expressed through their impact on percentage of defects introduced by developers who had made significant social and code communication (hero developers) vs who had made insignificant social and code communication (non-hero developers). The established wisdom in literature expresses that communication between developers is a key aspect of a successful project and also the established wisdom in the literature is to depreciate “heroes”, i.e., a small percentage of the staff who are responsible for most of the progress on a project. As James Herbsleb [1], in his ICSE’14 keynote said - “If two code sections communicate but the programmers of those two sections do not, then that code section is more likely to be buggy.”. Then the question arises, what is the effect on high or low social and code communication have on code quality, that is to say what effect of high social and code communication (i.e, heroism) have on code quality? Based on our study of 1000+ open source GitHub projects, we assert:

• •

Overwhelmingly, most projects have highly centralized social and code communication structure. That basically means, in majority of the projects there exists only a few developers who are responsible for majority of social and code communication.

• •

Developers who are responsible for majority of the social and code communication are far less likely to introduce bugs into the codebase than their counterparts. Thus showing developers who are responsible for majority of the social and code communication (that is hero developers) are beneficial towards the project quality and both social and code communication between developers have significant effect on code quality.

• •

Our experiments empirically supports the Herbsleb Hypothesis, showing - “If two code sections communicate but the programmers of those two sections do not, then that code section is more likely to be buggy.”.

One strange feature of our results is that what is old is now new. Our results (that heroes are important) echo a decade-old concept. In 1975, Fred Brooks wrote of “surgical teams” and the “chief programmer”  [58]. He argued that:

• •

Much as a surgical team during surgery is led by one surgeon performing the most critical work while directing the team to assist with less critical parts.

• •

Similarly, software projects should be led by one or a few “chief programmer” to develop critical system components while the rest of a team provides what is needed at the right time.

Brooks conjecture that “good” programmers are generally much more as productive as mediocre ones. This can be seen in the results that programmers are much more productive (who are responsible for majority of the social and code communication) and less likely to introduce bugs into the codebase. These developers are born when they become so skilled at what they do, that they assume a central position in a project. In our view, organizations need to acknowledge their dependency on such developers, perhaps altering their human resource policies and manage these people more efficiently by retaining them.

Also, our results suggest one other way to the way we develop new technologies or modern software projects. Specifically, given the prominence and importance of social and code communication between developers, future work could usefully explore methods to streamline the communication between developers that are critical for high quality software.

Finally, while this paper has been about how. social and code communication affects code quality, there might be a more general point to be made here: it is time to reflect more on long-held truisms in our field. While Heroes (developers who are responsible for majority of the social and code communication) are widely deprecated in the literature, yet empirically they are quite beneficial. What other statements in the literature need to be reviewed and revised?

Acknowledgements

This research was partially funded by NSF Grant #1908762.