On the Diversity of Software Package Popularity Metrics: An Empirical Study of npm

TL;DR

This study investigates the diverse ways to measure software package popularity on npm, revealing that different metrics often capture unrelated aspects and emphasizing the need for a unified framework for measuring popularity.

Contribution

It provides an empirical analysis of nine different popularity metrics across three sources, highlighting their differences and the necessity for a comprehensive measurement framework.

Findings

Popularity metrics are often unrelated and context-specific.

Different sources produce diverse popularity measures.

A large dataset of 175k npm packages was analyzed.

Abstract

Software systems often leverage on open source software libraries to reuse functionalities. Such libraries are readily available through software package managers like npm for JavaScript. Due to the huge amount of packages available in such package distributions, developers often decide to rely on or contribute to a software package based on its popularity. Moreover, it is a common practice for researchers to depend on popularity metrics for data sampling and choosing the right candidates for their studies. However, the meaning of popularity is relative and can be defined and measured in a diversity of ways, that might produce different outcomes even when considered for the same studies. In this paper, we show evidence of how different is the meaning of popularity in software engineering research. Moreover, we empirically analyse the relationship between different software popularity measures. As a case study, for a large dataset of 175k npm packages, we computed and extracted 9 different popularity metrics from three open source tracking systems: libraries.io, npmjs.com and GitHub. We found that indeed popularity can be measured with different unrelated metrics, each metric can be defined within a specific context. This indicates a need for a generic framework that would use a portfolio of popularity metrics drawing from different concepts.

Figures

Tables

Table 1. TABLE I: Descriptive summary of npm package popularity metrics.

Metric	Source	Description
# runtime dependent repositories	libraries.io	number of non-npm repositories that have a direct runtime dependency on the package
# transitive runtime dependents	libraries.io	number of npm packages that have a transitive runtime dependency on the package
# (direct) runtime dependents	libraries.io npm	number of npm packages that have a direct runtime dependency on the package
# downloads	npm	number of downloads of the package during the last year
# npm stars	npm	number of stars given by npm developers (https://docs.npmjs.com/cli/star)
# github stars	GitHub	number of stars of the package repository
# forks	GitHub	number of forks of the package repository
# pull requests	GitHub	number of pull requests and issues of the package repository
# subscribers	GitHub	number of subscribers/watchers of the package repository

Table 2. TABLE II: Pearson and Spearman correlation coefficients for libraries.io popularity metrics

	# transitive runtime dependents	# (direct) runtime dependents
# dependent runtime repositories	$\rho =0.63$ moderately strong $R=0.58$ moderate	$\rho =0.53$ moderate $R=0.71$ moderately strong
# direct runtime dependents	$\rho =0.66$ $R=0.62$ moderately strong

Table 3. TABLE III: Pearson and Spearman correlation coefficients for npm popularity metrics.

	# downloads	# (direct) runtime dependents
# npm stars	$\rho =0.39$ $R=0.33$ weak	$\rho =0.27$ weak $R=0.75$ moderately strong
# (direct) runtime dependents	$\rho =0.42$ $R=0.45$ moderate

Table 4. TABLE IV: Pearson and Spearman correlation coefficients for GitHub popularity metrics.

	#github stars	#forks	#subscribers
#pull requests	$\rho =0.64$  $R=0.64$ moderately strong	$\rho =0.7$  $R=0.63$ moderately strong	$\rho =0.53$  $R=0.51$ moderate
#subscribers	$\rho =0.55$  moderate  $R=0.7$  moderately strong	$\rho =0.55$ moderate  $R=0.68$ moderately strong
#forks	$\rho =0.73$  moderately strong  $R=0.85$  strong

Equations

$$\text{Aggarwal-Popularity}=\#\text{forks}+\#\text{stars}+\#{\text{pull requests}}^2$$

Full text

I Introduction

Depending on reusable software libraries is a common practice in software development [1]. It enables software developers to benefit from the functionality and maturity offered by these libraries, rather than needing to reimplement it from scratch. During the last years, the availability of very large distributions of open source projects in many domains has greatly increased this practice. Many programming languages feature at least one package management system to facilitate the use of such projects as external libraries [2]. Such package managers automate the distribution, installation and upgrading of thousands of different software packages [3].

For JavaScript, npm is by far the largest package manager in terms of number of hosted packages. Being so large, different packages frequently provide the same or similar functionality, making it challenging for developers to find and select the most appropriate package for their needs [4]. To face this challenge, a strategy that has been reported to be followed by developers is to use the package popularity as a main factor for its selection [5, 6]. Unfortunately, the concept of popularity can be interpreted very broadly, and can be measured in different ways.

Researchers rely on popularity metrics in order to sample their analyzed datasets or study software characteristics. Different studies rely on different popularity metrics, possibly producing different or even contradicting outcomes. A frequent way to characterise popularity is in terms of social aspects. For example, famous developers (so-called “rock stars”) have been shown to have a larger influence on where their followers contribute to than ordinary developers have on their followers [6]. Moreover, when Bogart et al. [5] interviewed OSS developers involved about the reasons behind selecting the appropriate dependencies for their software projects, most of the responses belonged to categories related to popularity and community reputation. Borges [7] found through a survey with 400 Stack Overflow users that the number of stars, forks and watchers of GitHub projects are all considered to be very useful popularity metrics.

Popularity can also be characterized in terms of technical aspects, such as use (e.g., number of downloads) or reuse (e.g., number of depending packages). Such definitions are based on the assumption that, if a package is widely (re)used, it can be trusted to be a good package.

The goal of our work is therefore to raise awareness of the relationship between, and the risk of using, different software package popularity measurements. As a case study, we focus on a large set of 175k npm packages, and analyse 9 different popularity metrics extracted from 3 different sources (libraries.io, npmjs.com and GitHub). We focus on one main research question: How are metrics of package popularity related to each other?

II Motivation and State of the Art

Popularity of open source software has been the subject of study, and used as basis for many empirical studies. In this section, we show how broad and different is the concept of popularity among researchers.

Capra et al. [8] studied the impact of company participation on popularity and software design quality for 643 SourceForge projects. Using popularity as the ranking index in SourceForge, they found that company involvement improves the popularity of open source projects. Borges et al. [9] measured popularity as the number of stars of GitHub projects. They analyzed the factors that impact this popularity, and also studied the impact of new features on project popularity. Analyzing 2,279 popular GitHub repositories, they identified four main patterns of popularity growth. Sajnani et al. [10] studied the relation between quality and library popularity in Maven, and considered popularity as the external usage across a set of open source Java projects.

Inspired by the work of Dabbish et al. [11], Karan et al. [12] investigated the relation between project popularity in GitHub and its consistent documentation updates. They considered popularity in terms of community interest, computed as the sum of the number of stars, number of forks and number of pull requests. Using this metric they found strong indicators that consistently popular projects exhibited consistent documentation effort. Syed et al. [13] analyzed the relation between the project popularity in terms of forks, watchers, stars, pull requests and code change frequency in GitHub projects. They showed that projects with at least 1,500 watchers each month have a strong positive correlation between project popularity and frequency of code changes. Kula et al. [14] considered library popularity as the number of internal dependents (i.e., internal usage: how many other libraries are using it), and proposed a model to visualize library popularity.

Dey et al. [15] analyzed 13k “popular” packages in npm, using linear regression and random forest models to inspect the effects of predictors representing different aspects of the software dependency supply chain on changes in numbers of downloads for a package. Considering popularity as downloads, they found that the number of downloads of upstream and downstream runtime dependencies have a strong effect on the number of package downloads. This suggests that, in order to interpret the package downloads properly, one should take into account the peculiarities of both upstream and downstream dependencies of that package.

The findings of the above studies are difficult to compare because they use different metrics to evaluate popularity. To our knowledge, there is no research focusing on how popularity measurements coming from different sources are related. The vision of this work is to raise awareness about the usage of, and relation between, different measurements of popularity. By doing so, we hope to gain more insights about how to understand and interpret the results of similar studies using popularity in different contexts.

III Method

The first step for studying popularity of software packages is to choose a relevant package management system. We require a widely used and well-known packaging system that involves a large and active developer community, so that measures about package popularity are relevant. We also need a large number of software packages, to minimize possible bias. These requirements led us to select npm, the largest packaging system both in number of packages (over $800k$ packages as of October 2018). To calculate popularity metrics for npm packages, we extracted and combined information from three online data sources and tracking services:

libraries.io, an open source repository containing metadata of package dependencies extracted from 36 package managers.111https://zenodo.org/record/1196312 (CC Share-Alike 4.0 license) The extracted metrics are based on the dataset of 13th March 2018, containing $698k$ packages. 2. 2.

npmjs.com, the official website for the npm package manager, allows to search for npm packages and sort them based on their popularity, quality or maintenance characteristics.222See https://docs.npmjs.com/getting-started/searching-for-packages. Popularity is computed as a weighted sum of number of stars, number of forks, number of subscribers, number of contributors, number of downloads, downloads acceleration and number of dependents. 3. 3.

GitHub, the hosting platform of many npm packager repositories. 73% of all npm packages are hosted on GitHub, 0.83% on Bitbucket, 0.69% on Gitlab, and 24.17% using an undefined repository. Thus, we decided to focus only on npm packages hosted on GitHub.

Starting from the libraries.io dataset, we restrict our analysis to packages that are at least two years old, because we consider that those packages had enough time to accumulate popularity. Then, using package names we extracted package information from npmjs.com. When different packages have the same GitHub repository (e.g., lodash and lodash-es), we choose the popularity statistics of the package with the highest number of downloads. This occurs in 6,898 distinct packages that share 2,336 GitHub repositories. This led us to a final set of 175,774 npm packages for our empirical analysis, whose data was extracted by April 2018.

IV How Are Metrics of Package Popularity Related to Each Other?

IV-A Dataset Exploration

Table I summarises the popularity metrics extracted from the different data sources used in our analysis. For the dependency-based metrics, we only considered runtime dependencies, i.e., those required to install and execute the package. Indeed, when calculating the number of (direct) runtime dependents for packages from libraries.io, we obtained almost identical values as the number of dependents reported by npmjs.com (with a linear correlation of $R=0.99$). The support team of npm indeed confirmed that the metric used by npmjs.com is based only on direct runtime dependencies, i.e., packages that are used as development or other dependencies are not taken into account333For more details on dependency types: https://docs.npmjs.com/files/package.json.

Exploring the dataset, we found that 27% of the npm packages have no direct runtime dependents, and 62% are not used in any external repository. We also found that only 39% of all packages have been downloaded more than 1,000 times in the last year and 76% have no npm stars. For the metrics from GitHub, we found that 32% of npm package repositories have no stars, 50% have no forks, 54% have only one subscriber and 43% of the repositories have no pull requests. This suggests that “popularity” can be quantified in many ways, obtaining different results.

To study the relation between different metrics, we perform a pairwise Pearson ($R$) and Spearman ($\rho$) correlation for all pairs of metrics. We use the following thresholds to interpret the absolute values of the correlation coefficients: $0<\textit{very\> weak}\leq 0.2<\textit{weak}\leq 0.4<\textit{moderate}\leq 0.6<\textit{moderately strong}\leq 0.8<\textit{strong}\leq 1$.

IV-B Metrics Emanating From the Same Source

To study how different metrics of popularity are related we consider the sources from where they were extracted as categories, and we study the correlation between different metrics from the same category.

IV-B1 libraries.io

Dependency-based metrics can be used to determine if a package is a top-level application dependency (i.e., used often by external repositories), or a popular internal library (i.e., used often by other npm packages). Table II reports the correlation results for the considered pairs of metrics from libraries.io. The correlation results vary from moderate to moderately strong. This means that choosing one of these metrics as the popularity measure will provide similar, but slightly different results than the other metrics.

IV-B2 npm

Between the metrics extracted from npm there was little correlation (see Table III). We found a moderately strong Pearson correlation between # npm stars and # runtime dependents but the Spearman correlation was weak. The other correlations were weak to moderate.

IV-B3 GitHub

Metrics from GitHub are important indicators of how the community of developers accepts and judges a software project. Similar to Borges et al. [9] analysis of the factors that impact the # github stars, Table IV shows a strong correlation with the # forks of the package repositories. For all pairs of GitHub popularity metrics, we found a moderate to moderately strong correlation, except for the pair of (#subscribers, #pull requests), where we only found moderate Spearman and Pearson correlations.

IV-C Metrics Emanating from Different Sources

To limit the number of popularity metrics to consider, we focus on 6 metrics: #dependent repositories from libraries.io, all three metrics from npmjs.com, #subscribers and a single popularity metric –defined and used by Aggarwal et al. [12]– that combines the three GitHub metrics:

[TABLE]

Figure 1 reports the Spearman correlation between these 6 metrics, revealing that the highest correlation is only moderate ($\rho=0.54$), for the pair (Aggarwal-Popularity, #downloads).

To study the relation between the 6 metrics in more detail we use the # direct runtime dependents metric as a main sorting criterion to select a set of 1,000 packages out of our larger dataset, thereby simulating a search on the npm home page that by default retrieves the most depended upon packages444https://www.npmjs.com/browse/depended. For the other 5 popularity metrics we compute the sets of top 1,000 packages in a similar way, and we compare all possible intersections between these sets and the one based on the main sorting criterion.

The results are shown using a Venn diagram in Figure 2. If we consider the intersection of all popularity metrics, we find only 5.1% (i.e., 51 out of 1,000) packages. If we compare the # direct runtime dependents metric with each of the 5 reported metrics individually, we obtain the largest intersections with #npm stars (601 packages), and #dependent repositories (549 packages). The smallest intersection is found for #subscribers (166 packages) and Aggarwal-Popularity (194 packages), that have less than 20% in common with the # direct runtime dependents selection. The reported numbers are in general low, indicating that the main sorting criterion generates many different packages than the other popularity metrics.

V Discussion

Our empirical analysis revealed that most of the considered popularity metrics are not strongly correlated to each other. Even if they are, as was the case for # forks versus # github stars (see Table IV), other authors have found lower correlations. For example, [7] only found a moderate correlation ($\rho=0.558$), showing the influence of the chosen population555They relied on GitHub’s 5000 most starred repositories. on the outcome of the results.

Popularity is indeed relative and should be always measured to a specific context. This finding should be taken into account when designing automated recommendation tools and search engines for software libraries. For example, the npms.io search engine for npm packages, can sort results based on popularity, quality, maintenance, or a combination of those. The popularity search criterion is based on a weighted sum of 7 more primitive metrics. Using other weights or alternative definitions of popularity may produce quite different popularity rankings.

Implementations of popularity metrics might also produce incorrect values, e.g., because they were computed incorrectly, or because the data has been “violated” by spammers. For example, in 2016 an application was created666https://github.com/ell/npm-gen-all to generate many spam npm packages (e.g., neat-x and wowdude-x777https://libraries.io/npm/wowdude-119) that used a huge number of dependencies. This affected the # downloads and # dependent packages for those packages that were used as dependencies. Software maintainers are raising awareness about such download inflation and related issues, e.g., “What happens if a large team is using the library and now has to download it for every single one of their users? More download inflation.”[16].

The above phenomena may impact reproducibility of research studies if the population of software libraries under analysis is retrieved using a single metric of popularity (e.g., [7] analysed the top 5,000 GitHub projects based on their number of stars). Other popularity metrics may lead to different research findings, limiting the generalisability of the results.

All of the above issues are important to consider if the goal is to use or analyse popularity metrics across software libraries with different characteristics. Thus, there is an urgent need for a popularity measurement framework including a wide range of popularity metrics, as previous studies have done with other open source software characteristics (e.g., software reusability [17], software quality [18] and software success [19]).

Our findings are valuable for the open source community at large. For instance, the Linux Foundation’s CHAOSS project focuses on creating analytics and metrics to help define community health. One of its goals is to define metrics for measuring community activity, contributions, and health, and eventually produce detailed use cases or recommendations to analyze specific issues in the industry/OSS world888https://chaoss.community/metrics/. Our analysis could be a contribution to the project in order to define the most important metrics for popularity.

VI Threats to Validity

The reported correlation results may be biased by the selected dataset, and the filtering to only npm packages hosted on GitHub with at least two years of history. To limit this bias we analyzed more than 175k npm packages. Our results may not be generalizable to JavaScript packages that are not distributed through npm, that are not hosted on GitHub, or that are published by other software package managers. We cannot generalise either to packages that are developed in other programming languages.

Another threat to generalisability is that we have restricted the dependency analysis to runtime dependencies only. Hence, the popularity of packages that are frequently or exclusively used as development dependencies has been underestimated.

Many other popularity metrics could have been considered in the study (e.g., libraries.io SourceRank999https://github.com/librariesio/libraries.io/issues/1916). Not including them in the analysis may have influenced the results of our analysis.

VII Conclusion and Future Work

Software popularity is an important indicator of software success. It can be a main factor for gaining more attention and adoption, and it may attract new developers to the project. However, which characteristic of software popularity to be considered should be defined by the specific context of use.

In this paper, we showed that popularity can be measured in many different ways, and researchers have studied popularity using a wide variety of different metrics. Their research findings may depend on the metrics they used. To illustrate this, we relied on a large dataset of JavaScript packages in npm to study the correlation between different popularity metrics used by different open source services. We observed that many popularity metrics are not strongly correlated, implying that the use of different metrics may produce different outcomes. This calls for the definition and use of a measurement framework that takes into account the diversity and context-dependence of software popularity.

As future work, we plan to complement our quantitative analysis with qualitative interviews of how developers actually use popularity measures, and whether they are concerned about the issues that relying on such measures may bring. The outcome of such an analysis may ultimately lead to better services for searching and recommending software libraries.

We also plan to reproduce existing research studies in order to assess to which extent the use of specific popularity metrics affects the research findings. In a similar vein, we aim to extend our analysis to package managers for other programming languages (e.g., PyPI for Python, Maven for Java, RubyGems for Ruby).

Acknowledgment

This work was partially supported by the EU Research FP (H2020-MSCA-ITN-2014-642954, Seneca), the Spanish Government (TIN2014-59400-R, SobreVision), the Excellence of Science Project SECO-Assist (O015718F, FWO - Vlaanderen and F.R.S.-FNRS).