Characterizing Deep Learning Package Supply Chains in PyPI: Domains, Clusters, and Disengagement

TL;DR

This study analyzes PyPI's deep learning package supply chains, revealing their domain distributions, cluster structures, and reasons for package disengagement, providing insights into dependency management and maintenance practices.

Contribution

It offers a comprehensive characterization of DL package supply chains in PyPI, including domain coverage, cluster shapes, and disengagement reasons, based on large-scale metadata analysis.

Findings

Popular packages cover 34 domains across 8 categories.

Clusters mainly exhibit Arrow and Star shapes, with Tree and Forest being more complex.

Disengagement reasons include dependency issues, functional improvements, and installation ease.

Abstract

Deep learning (DL) package supply chains (SCs) are critical for DL frameworks to remain competitive. However, vital knowledge on the nature of DL package SCs is still lacking. In this paper, we explore the domains, clusters, and disengagement of packages in two representative PyPI DL package SCs to bridge this knowledge gap. We analyze the metadata of nearly six million PyPI package distributions and construct version-sensitive SCs for two popular DL frameworks: TensorFlow and PyTorch. We find that popular packages (measured by the number of monthly downloads) in the two SCs cover 34 domains belonging to eight categories. Applications, Infrastructure, and Sciences categories account for over 85% of popular packages in either SC and TensorFlow and PyTorch SC have developed specializations on Infrastructure and Applications packages respectively. We employ the Leiden community detection algorithm and detect 131 and 100 clusters in the two SCs. The clusters mainly exhibit four shapes: Arrow, Star, Tree, and Forest with increasing dependency complexity. Most clusters are Arrow or Star, but Tree and Forest clusters account for most packages (Tensorflow SC: 70%, PyTorch SC: 90%). We identify three groups of reasons why packages disengage from the SC (i.e., remove the DL framework and its dependents from their installation dependencies): dependency issues, functional improvements, and ease of installation. The most common disengagement reason in the two SCs are different. Our study provides rich implications on the maintenance and dependency management practices of PyPI DL SCs.