Exploiting Leaderboards for Large-Scale Distribution of Malicious Models

TL;DR

This paper reveals how adversaries can exploit model leaderboards to distribute poisoned models at scale, demonstrating a framework that embeds malicious behaviors while maintaining high leaderboard rankings across multiple modalities.

Contribution

The paper introduces TrojanClimb, a novel framework enabling malicious model injection that preserves leaderboard performance, exposing a critical security vulnerability in model evaluation platforms.

Findings

Adversaries can achieve high leaderboard rankings with poisoned models.

TrojanClimb is effective across text, speech, and image modalities.

Leaderboard systems are vulnerable to large-scale malicious model distribution.

Abstract

While poisoning attacks on machine learning models have been extensively studied, the mechanisms by which adversaries can distribute poisoned models at scale remain largely unexplored. In this paper, we shed light on how model leaderboards -- ranked platforms for model discovery and evaluation -- can serve as a powerful channel for adversaries for stealthy large-scale distribution of poisoned models. We present TrojanClimb, a general framework that enables injection of malicious behaviors while maintaining competitive leaderboard performance. We demonstrate its effectiveness across four diverse modalities: text-embedding, text-generation, text-to-speech and text-to-image, showing that adversaries can successfully achieve high leaderboard rankings while embedding arbitrary harmful functionalities, from backdoors to bias injection. Our findings reveal a significant vulnerability in the machine learning ecosystem, highlighting the urgent need to redesign leaderboard evaluation mechanisms to detect and filter malicious (e.g., poisoned) models, while exposing broader security implications for the machine learning community regarding the risks of adopting models from unverified sources.