On the Suitability of Hugging Face Hub for Empirical Studies

TL;DR

This paper explores the potential of Hugging Face Hub as a data source for empirical software engineering studies, comparing its features and analyzing its data to assess its suitability.

Contribution

It provides an initial evaluation of Hugging Face Hub's features and data for empirical research, filling a gap in understanding its research potential.

Findings

HFH has unique features compared to GitHub and GitLab.

The data in HFH shows promise for empirical studies.

Further analysis is needed to confirm its research suitability.

Abstract

Background. The development of empirical studies in software engineering mainly relies on the data available on code hosting platforms, being GitHub the most representative. Nevertheless, in the last years, the emergence of Machine Learning (ML) has led to the development of platforms specifically designed for developing ML-based projects, being Hugging Face Hub (HFH) the most popular one. With over 250k repositories, and growing fast, HFH is becoming a promising ecosystem of ML artifacts and therefore a potential source of data for empirical studies. However, so far there have been no studies evaluating the potential of HFH for such studies. Objective. In this proposal for a registered report, we aim at performing an exploratory study of the current state of HFH in order to investigate its suitability to be used as a source platform for empirical studies. Method. We conduct a qualitative and quantitative analysis of HFH for empirical studies. The former will be performed by comparing the features of HFH with those of other code hosting platforms, such as GitHub and GitLab. The latter will be performed by analyzing the data available in HFH.