Queries, Representation & Detection: The Next 100 Model Fingerprinting Schemes

TL;DR

This paper systematically analyzes model fingerprinting schemes by decomposing them into Query, Representation, and Detection components, revealing that simple baselines perform comparably to complex methods and highlighting the need for more challenging benchmarks.

Contribution

It introduces a systematic framework for creating and evaluating model fingerprinting schemes, uncovering numerous unexplored combinations and providing open-source tools for future research.

Findings

Simple baseline performs on par with complex fingerprints

Many unexplored QuRD combinations identified

Highlights need for more challenging benchmarks

Abstract

The deployment of machine learning models in operational contexts represents a significant investment for any organisation. Consequently, the risk of these models being misappropriated by competitors needs to be addressed. In recent years, numerous proposals have been put forth to detect instances of model stealing. However, these proposals operate under implicit and disparate data and model access assumptions; as a consequence, it remains unclear how they can be effectively compared to one another. Our evaluation shows that a simple baseline that we introduce performs on par with existing state-of-the-art fingerprints, which, on the other hand, are much more complex. To uncover the reasons behind this intriguing result, this paper introduces a systematic approach to both the creation of model fingerprinting schemes and their evaluation benchmarks. By dividing model fingerprinting into three core components -- Query, Representation and Detection (QuRD) -- we are able to identify $\sim100$ previously unexplored QuRD combinations and gain insights into their performance. Finally, we introduce a set of metrics to compare and guide the creation of more representative model stealing detection benchmarks. Our approach reveals the need for more challenging benchmarks and a sound comparison with baselines. To foster the creation of new fingerprinting schemes and benchmarks, we open-source our fingerprinting toolbox.