npm-follower: A Complete Dataset Tracking the NPM Ecosystem

TL;DR

npm-follower is a comprehensive dataset that archives all NPM packages and versions, including deleted ones, enabling researchers to analyze the entire ecosystem's structure, security, and malware concerns.

Contribution

It introduces a novel crawling architecture that captures and retains all package data from NPM, including deleted versions, addressing limitations of prior datasets.

Findings

Includes over 35 million package versions

Captures data on deleted packages and versions

Grows at about 1 million versions per month

Abstract

Software developers typically rely upon a large network of dependencies to build their applications. For instance, the NPM package repository contains over 3 million packages and serves tens of billions of downloads weekly. Understanding the structure and nature of packages, dependencies, and published code requires datasets that provide researchers with easy access to metadata and code of packages. However, prior work on NPM dataset construction typically has two limitations: 1) only metadata is scraped, and 2) packages or versions that are deleted from NPM can not be scraped. Over 330,000 versions of packages were deleted from NPM between July 2022 and May 2023. This data is critical for researchers as it often pertains to important questions of security and malware. We present npm-follower, a dataset and crawling architecture which archives metadata and code of all packages and versions as they are published, and is thus able to retain data which is later deleted. The dataset currently includes over 35 million versions of packages, and grows at a rate of about 1 million versions per month. The dataset is designed to be easily used by researchers answering questions involving either metadata or program analysis. Both the code and dataset are available at https://dependencies.science.