Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models

TL;DR

Open-source machine learning models offer significant benefits but pose substantial privacy and security risks, including hidden functionalities and potential malicious manipulations, which require increased awareness and responsible use.

Contribution

This paper provides a comprehensive overview of privacy and security threats in open-source AI models, highlighting risks and promoting responsible AI deployment.

Findings

Open-source models can conceal malicious functionalities.

Security breaches can lead to physical harm or data leaks.

Awareness is crucial for safe AI adoption.

Abstract

The field of artificial intelligence (AI) has experienced remarkable progress in recent years, driven by the widespread adoption of open-source machine learning models in both research and industry. Considering the resource-intensive nature of training on vast datasets, many applications opt for models that have already been trained. Hence, a small number of key players undertake the responsibility of training and publicly releasing large pre-trained models, providing a crucial foundation for a wide range of applications. However, the adoption of these open-source models carries inherent privacy and security risks that are often overlooked. To provide a concrete example, an inconspicuous model may conceal hidden functionalities that, when triggered by specific input patterns, can manipulate the behavior of the system, such as instructing self-driving cars to ignore the presence of other vehicles. The implications of successful privacy and security attacks encompass a broad spectrum, ranging from relatively minor damage like service interruptions to highly alarming scenarios, including physical harm or the exposure of sensitive user data. In this work, we present a comprehensive overview of common privacy and security threats associated with the use of open-source models. By raising awareness of these dangers, we strive to promote the responsible and secure use of AI systems.