A Rusty Link in the AI Supply Chain: Detecting Evil Configurations in Model Repositories

TL;DR

This paper investigates security vulnerabilities in AI model repositories, especially malicious configurations on Hugging Face, and introduces CONFIGSCAN, an LLM-based tool for detecting suspicious configurations with high accuracy.

Contribution

It provides the first comprehensive analysis of malicious configurations in AI repositories and proposes CONFIGSCAN, a novel LLM-based detection method for security threats.

Findings

Thousands of suspicious repositories identified

CONFIGSCAN achieves low false positives and high detection accuracy

Highlights urgent need for security validation in AI hosting platforms

Abstract

Recent advancements in large language models (LLMs) have spurred the development of diverse AI applications from code generation and video editing to text generation; however, AI supply chains such as Hugging Face, which host pretrained models and their associated configuration files contributed by the public, face significant security challenges; in particular, configuration files originally intended to set up models by specifying parameters and initial settings can be exploited to execute unauthorized code, yet research has largely overlooked their security compared to that of the models themselves; in this work, we present the first comprehensive study of malicious configurations on Hugging Face, identifying three attack scenarios (file, website, and repository operations) that expose inherent risks; to address these threats, we introduce CONFIGSCAN, an LLM-based tool that analyzes configuration files in the context of their associated runtime code and critical libraries, effectively detecting suspicious elements with low false positive rates and high accuracy; our extensive evaluation uncovers thousands of suspicious repositories and configuration files, underscoring the urgent need for enhanced security validation in AI model hosting platforms.