SourceBroken: A large-scale analysis on the (un)reliability of SourceRank in the PyPI ecosystem

TL;DR

This study critically evaluates the reliability of SourceRank in the PyPI ecosystem, revealing its vulnerability to evasion attacks like URL confusion and its inability to reliably distinguish malicious packages in real-world data.

Contribution

The paper provides a comprehensive threat model for SourceRank, analyzes its effectiveness against evasion attacks, and demonstrates its limitations in real-world scenarios.

Findings

SourceRank fails to reliably distinguish malicious packages in real-world data.

URL confusion is an emerging attack vector increasing in prevalence.

SourceRank's failure to reflect package removals limits its usefulness.

Abstract

SourceRank is a scoring system made of 18 metrics that assess the popularity and quality of open-source packages. Despite being used in several recent studies, none has thoroughly analyzed its reliability against evasion attacks aimed at inflating the score of malicious packages, thereby masquerading them as trustworthy. To fill this gap, we first propose a threat model that identifies potential evasion approaches for each metric, including the URL confusion technique, which can affect 5 out of the 18 metrics by leveraging a URL pointing to a legitimate repository potentially unrelated to the malicious package. Furthermore, we study the reliability of SourceRank in the PyPI ecosystem by analyzing the SourceRank distributions of benign and malicious packages in the state-of-the-art MalwareBench dataset, as well as in a real-world dataset of 122,398 packages. Our analysis reveals that, while historical data suggests a clear distinction between benign and malicious packages, the real-world distributions overlap significantly, mainly due to SourceRank's failure to timely reflect package removals. As a result, SourceRank cannot be reliably used to discriminate between benign and malicious packages in real-world scenarios, nor to select benign packages among those available on PyPI. Finally, our analysis reveals that URL confusion represents an emerging attack vector, with its prevalence increasing from 4.2% in MalwareBench to 7.0% in our real-world dataset. Moreover, this technique is often used alongside other evasion techniques and can significantly inflate the SourceRank metrics of malicious packages.