Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software

TL;DR

This study investigates how open-source projects on GitHub use ML models, assessing decision scope, safety measures, and compliance with terms of use to inform guidelines and detection tools.

Contribution

It provides early evidence of non-compliant ML model usage in open-source software and proposes groundwork for regulatory guidelines and automated violation detection.

Findings

Many projects use ML models for decision-making without safety measures

Some models are used in ways that may violate terms of use

The study highlights the need for guidelines and automated compliance tools

Abstract

The increasing availability of Machine Learning (ML) models, particularly foundation models, enables their use across a range of downstream applications, from scenarios with missing data to safety-critical contexts. This, in principle, may contravene not only the models' terms of use, but also governmental principles and regulations. This paper presents a preliminary investigation into the use of ML models by 173 open-source projects on GitHub, spanning 16 application domains. We evaluate whether models are used to make decisions, the scope of these decisions, and whether any post-processing measures are taken to reduce the risks inherent in fully autonomous systems. Lastly, we investigate the models' compliance with established terms of use. This study lays the groundwork for defining guidelines for developers and creating analysis tools that automatically identify potential regulatory violations in the use of ML models in software systems.