I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks

TL;DR

This paper provides standardized guidelines and a comprehensive framework for designing, evaluating, and comparing model stealing attacks, addressing a key gap in the security assessment of machine learning models.

Contribution

It introduces the first comprehensive threat model and evaluation framework for model stealing attacks on image classification models, along with best practices and open research questions.

Findings

Developed a standardized threat model for model stealing attacks

Created a framework for attack comparison and evaluation

Identified key tasks and models most studied in the field

Abstract

Model stealing attacks endanger the confidentiality of machine learning models offered as a service. Although these models are kept secret, a malicious party can query a model to label data samples and train their own substitute model, violating intellectual property. While novel attacks in the field are continually being published, their design and evaluations are not standardised, making it challenging to compare prior works and assess progress in the field. This paper is the first to address this gap by providing recommendations for designing and evaluating model stealing attacks. To this end, we study the largest group of attacks that rely on training a substitute model -- those attacking image classification models. We propose the first comprehensive threat model and develop a framework for attack comparison. Further, we analyse attack setups from related works to understand which tasks and models have been studied the most. Based on our findings, we present best practices for attack development before, during, and beyond experiments and derive an extensive list of open research questions regarding the evaluation of model stealing attacks. Our findings and recommendations also transfer to other problem domains, hence establishing the first generic evaluation methodology for model stealing attacks.

Tables

Table 1. Table 1 . Threat models in prior works. Attacker’s knowledge: data (first column), target outputs (column ”Outputs”: ✢ - labels, ✻ - confidence scores, ❋ - explanations or gradients), and target architecture (column ”Model”: ■ \blacksquare  if target and substitute are different, □ \square   – if the same). Attacker’s capabilities: maximum number of queries (column ”QB”: - 10k, - 10k-100k, - 100k-1m, - 1m). Attacker’s goal: evaluation metrics (column ”Metrics”: A - accuracy, F - fidelity, T - transferability).

	Paper	Outputs	Model	QB	Metrics
Original data	Pengcheng et al. (Pengcheng et al., 2018)	-	$■$		AFT
	Papernot et al. (Papernot et al., 2017)	✢	$■$		AFT
	Pape et al. (Pape et al., 2023)	✢	$■$/$\square$	-	AFT
	Yan et al. (Yan et al., 2023c)	✢	$■$/$\square$		AFT
	Juuti et al. (Juuti et al., 2019)	✢/✻	$■$/$\square$		AFT
	Jagielski et al. (Jagielski et al., 2020)	✻	-		AFT
	Liu et al. (Liu et al., 2023)	✻	-		AFT
	Yu et al. (Yu et al., 2020)	✻	$■$		AFT
	Zhao et al. (Zhao et al., 2023)	✻	$■$		AFT
	Zhang et al. (Zhang et al., 2021)	✻	$■$/$\square$		AFT
	Chen et al. (Chen et al., 2023)	✻	$\square$		AFT
	He et al. (He et al., 2021b)	✻	$\square$		AFT
	Li et al. (Li et al., 2025)	✻	$\square$		AFT
	Liu et al. (Liu et al., 2022b)	✻	$\square$		AFT
	Yan et al. (Yan et al., 2023b)	❋	-		AFT
	Yan et al. (Yan et al., 2022)	❋	$■$	-	AFT
	Milli et al. (Milli et al., 2019)	❋	$■$/$\square$		AFT
PD data	Correia-Silva et al. (Correia-Silva et al., 2018)	✢	-		AFT
	Yu et al. (Yu et al., 2020)	✻	$■$		AFT
	Zhao et al. (Zhao et al., 2023)	✻	$■$		AFT
	Xie et al. (Xie et al., 2022)	✻	$■$		AFT
nPD data	Wang et al. (Wang and Lin, 2022)	✢	-		AFT
	Correia-Silva et al. (Correia-Silva et al., 2018)	✢	-		AFT
	Mosafi et al. (Mosafi et al., 2019)	✢	$■$	-	AFT
	Karmakar et al. (Karmakar and Basu, 2023)	✢	$■$		AFT
	Gong et al. (Gong et al., 2021)	✢	$■$		AFT
	Jindal et al. (Jindal et al., 2024)	✢	$■$		AFT
	Yan et al. (Yan et al., 2023c)	✢	$■$/$\square$		AFT
	Wang et al. (Wang et al., 2022)	✢	$■$/$\square$		AFT
	Yang et al. (Yang et al., 2024)	✢	$\square$		AFT
	Orekondy et al. (Orekondy et al., 2019)	✢/✻	$■$/$\square$		AFT
	Pal et al. (Pal et al., 2019, 2020)	✢/✻	$■$/$\square$		AFT
	Atli et al. (Atli et al., 2020)	✢/✻	$\square$		AFT
	Zhao et al. (Zhao et al., 2023)	✻	$■$		AFT
	Xie et al. (Xie et al., 2022)	✻	$■$		AFT
	Khaled et al. (Khaled et al., 2022)	✻	$■$		AFT
	Barbalau et al. (Barbalau et al., 2020)	✻	$■$/$\square$	-	AFT
	Chen et al. (Chen et al., 2023)	✻	$\square$		AFT
Data-free	Yang et al. (Yang et al., 2023b)	✢	$■$	-	AFT
	Sanyal et al. (Sanyal et al., 2022)	✢	$■$		AFT
	Hondru et al. (Hondru and Ionescu, 2025)	✢/✻	$■$		AFT
	Zhang et al. (Zhang et al., 2022a)	✢/✻	$■$		AFT
	Hong et al. (Hong et al., 2023)	✢/✻	$■$		AFT
	Beetham et al. (Beetham et al., 2023)	✢/✻	$■$		AFT
	Yuan et al. (Yuan et al., 2022)	✢/✻	$■$/$\square$	-	AFT
	Lin et al. (Lin et al., 2023)	✢/✻	$■$/$\square$		AFT
	Rosenthal et al. (Rosenthal et al., 2023)	✢/✻	$■$/$\square$		AFT
	Truong et al. (Truong et al., 2021)	✻	$■$		AFT
	Liu et al. (Liu et al., 2024)	✻	$■$		AFT
	Kariyappa et al. (Kariyappa et al., 2021)	✻	$■$		AFT
	Roberts et al. (Roberts et al., 2019)	✻	$\square$		AFT
	Miura et al. (Miura et al., 2021)	❋	$■$		AFT
	Yan et al. (Yan et al., 2023a)	❋	$■$		AFT

Table 2. Table 2 . Transferability measurement in related work with used attack and measured as perturbation size.

Paper	Adversarial algorithm	Max. perturbation
Beetham et al. (Beetham et al., 2023)	(Goodfellow et al., 2015) (FGSM), (Madry et al., 2018) (PGD)	0.01
Jindal et al. (Jindal et al., 2024)	(Madry et al., 2018) (PGD)	0.03
Wang et al. (Wang et al., 2022)	(Madry et al., 2018) (PGD)	0.03
Khaled et al. (Khaled et al., 2022)	(Goodfellow et al., 2015) (FGSM), (Madry et al., 2018) (PGD)	0.03, 0.05, 0.10, 0.15
Zhao et al. (Zhao et al., 2023)	(Goodfellow et al., 2015) (FGSM)	0.03, 0.06, 0.09, 0.12, 0.15, 0.18, 0.24
Zhang et al. (Zhang et al., 2022a)	(Goodfellow et al., 2015) (FGSM), (Kurakin et al., 2017) (BIM), (Madry et al., 2018) (PGD)	0.03, 0.13
Yuan et al. (Yuan et al., 2022)	(Madry et al., 2018) (PGD)	0.03, 0.3
Papernot et al. (Papernot et al., 2017)	(Goodfellow et al., 2015) (FGSM), (Papernot et al., 2016b)	0.05, 0.10, 0.20, 0.25, 0.30, 0.50, 0.70, 0.90
Pengcheng et al. (Pengcheng et al., 2018)	(Goodfellow et al., 2015) (FGSM)	0.2
Pal et al. (Pal et al., 2020)	(Goodfellow et al., 2015) (FGSM)	0.25
Juuti et al. (Juuti et al., 2019)	(Goodfellow et al., 2015) (FGSM)	0.25
He et al. (He et al., 2021b)	(Madry et al., 2018) (PGD)	0.5

Table 3. Table 3 . Frequency of usage of a specific architecture for training target models on various datasets. Some of the papers contribute with multiple entries.

	CIFAR-10	MNIST	FMNIST	SVHN	GTSRB	CIFAR-100	Others
ResNet34	17	2	2	11	1	4	16
ResNet18	6	1	3	1		3	4
LeNet		5	6				1
AlexNet	5		3				5
VGG16	3	2	3		1	1	2
VGG19	1	3	4			1	3
ResNet50	4	1	1			1	1
Others	24	25	22	4	7	13	12

Table 4. Table 4 . Frequency of usage of a specific architecture for training substitute models (rows) to attack various target models (columns). Some of the papers contribute with multiple entries.

	ResNet34	ResNet18	LeNet	AlexNet	VGG16	VGG19	ResNet50	Others
ResNet34	9	1			1		1	3
ResNet18	16	10		3	1		3	6
LeNet			5	2				2
AlexNet	1	2	2	4	1			3
VGG16	4		1		5		1	4
VGG11	2	2		2	2		1	6
RN50	2				1		1	7
half-AlexNet				7
Others	4		5		2	3	12	97

Table 5. Table 5 . Original datasets, target and substitute architectures per paper.

Paper	Original dataset	Target architecture	Substitute architecture
(Jagielski et al., 2020)	CIFAR-10, SVHN	WSL, WideResNet-28-2	ResNet-v2-50, ResNet-v2-200, N/A
(Papernot et al., 2017)	GTSRB, MNIST	N/A	Custom
(Correia-Silva et al., 2018)	AR Face, BU3DFE, JAFFE, MMI, RaFD, CIFAR-10	N/A	VGG-16
(Pal et al., 2020)	CIFAR10, GTSRB, MNIST	Custom	Custom
(Juuti et al., 2019)	GTSRB, MNIST	Custom	Custom
(Orekondy et al., 2019)	Caltech256, CUB-200-2011, Diabetic5, Indoor67	ResNet-34, VGG-16	AlexNet, DenseNet-161, ResNet-18, ResNet-34, ResNet-50, VGG-16
(Atli et al., 2020)	Caltech256, CIFAR-10, CUB-200-2011, Diabetic5, Indoor67	ResNet-34	ResNet-34
(Pengcheng et al., 2018)	CIFAR-10, FMNIST, MNIST	N/A, some ResNet	Custom, VGG-19
(Pal et al., 2019)	CIFAR-10, FMNIST, GTSRB, MNIST	Custom	Custom
(Mosafi et al., 2019)	CIFAR-10	Custom	VGG-16
(Yuan et al., 2022)	CIFAR-10, KMNIST, MNIST, SVHN	LeNet, ResNet-18, ResNet-34	LeNet, ResNet-18, ResNet-34
(Milli et al., 2019)	CIFAR-10, MNIST	Custom, MLogReg, ResNet-18, VGG-11	Custom, MLogReg, ResNet-18, VGG-11
(Kariyappa et al., 2021)	CIFAR-10, FMNIST, GTSRB, SVHN	LeNet, ResNet-20	WideResNet-22
(Roberts et al., 2019)	FMNIST, KMNIST, MNIST, notMNIST	Custom	Custom
(Barbalau et al., 2020)	CIFAR-10, FMNIST, 10 Monkey Species	AlexNet, LeNet, ResNet-18, VGG-16	half-AlexNet, half-LeNet, ResNet-18, VGG-16
(Yu et al., 2020)	GTSRB, VGG Flower	AlexNet, ResNet-50, VGG-19, VGG-Face	ResNet-50, VGG-19, VGG-Face
(Gong et al., 2021)	CIFAR-10, GTSRB, MNIST	Custom	Custom, ResNet-18
(Truong et al., 2021)	CIFAR-10, SVHN	ResNet-34	ResNet-18
(Miura et al., 2021)	CIFAR-10, SVHN	ResNet-34	ResNet-18
(Sanyal et al., 2022)	CIFAR-10, CIFAR-100	AlexNet, ResNet-18, ResNet-34	half-AlexNet, ResNet-18
(Zhang et al., 2021)	Caltech256, ImageNet, FMNIST	AlexNet, Custom, LeNet, ResNet-34, ResNet-50	AlexNet, Custom, LeNet, ResNet-18, ResNet-34, ResNet-50
(Wang and Lin, 2022)	Caltech256, CIFAR-10, CUB-200-2011, SVHN	ResNet-34	N/A
(Wang et al., 2022)	Caltech256, CIFAR-10, CUB-200-2011, SVHN	ResNet-34	some DenseNet, ResNet-18, ResNet-34, ResNet-50, VGG-16
(Yan et al., 2022)	CIFAR-10, CIFAR-100, FMNIST, MNIST	DenseNet-161, ResNet-50, VGG-19	Custom
(Xie et al., 2022)	BelgiumTSC, MNIST	AlexNet, LeNet	half-AlexNet, half-LeNet, ResNet-18, VGG-16
(Chen et al., 2023)	CIFAR-10, FMNIST, GTSRB, ImageNette, MNIST	LeNet, ResNet-34, VGG-16	LeNet, ResNet-34, VGG-16
(He et al., 2021b)	CIFAR-10, ImageNet, MNIST	Custom, Inception-V3, LeNet, ResNet-18, ResNet-152	Custom, Inception-V3, LeNet, ResNet-18, ResNet-152
(Liu et al., 2022b)	CelebA, FMNIST, STL-10, UTKFace	AlexNet, Custom, ResNet-18, VGG-19, Xception	AlexNet, Custom, ResNet-18, VGG-19, Xception
(Rosenthal et al., 2023)	CIFAR-10, CIFAR-100	ResNet-18, ResNet-34	ResNet-18
(Yan et al., 2023c)	CIFAR-10, CIFAR-100, FMNIST, MNIST	DenseNet-121, DenseNet-161, DenseNet-169, DenseNet-201, Inception-V1, Inception-V2, Inception-V3, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, VGG-11, VGG-13, VGG-16, VGG-19	DenseNet-121, DenseNet-161, DenseNet-169, DenseNet-201, Inception-V1, Inception-V2, Inception-V3, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, VGG-11, VGG-13, VGG-16, VGG-19
(Zhang et al., 2022a)	CIFAR-10, CIFAR-100, FMNIST, MNIST, SVHN, TinyImageNet	N/A, Custom, ResNet-34	N/A, Custom, ResNet-18
(Yan et al., 2023a)	CIFAR-10, SVHN	ResNet-34	ResNet-18
(Yan et al., 2023b)	CIFAR-10, CIFAR-100, FMNIST, MNIST	DenseNet-161, ResNet-50, VGG-19	N/A
(Yang et al., 2023b)	CIFAR-10, FMNIST, MNIST	AlexNet, LeNet	half-AlexNet, half-LeNet
(Lin et al., 2023)	CIFAR-10, FMNIST	AlexNet, LeNet, ResNet-18, VGG-11	AlexNet, ResNet-18, VGG-11
(Liu et al., 2023)	FMNIST, Intel-Image	N/A	SqueezeNet
(Pape et al., 2023)	CIFAR-10, SVHN	N/A, ResNet-152	Custom, Inception-V3, ResNet-152
(Liu et al., 2024)	CelebA, CIFAR-10, SVHN	ResNet-34	ResNet-18
(Hondru and Ionescu, 2025)	CIFAR-10, Food-101	AlexNet, ResNet-50	half-AlexNet, ResNet-18
(Khaled et al., 2022)	CIFAR-10, MNIST, SVHN	Custom, ResNet-34	LeNet, VGG-16
(Zhao et al., 2023)	CIFAR-10, Flower-17, GTSRB, STL-10	VGG-13	ResNet-50
(Karmakar and Basu, 2023)	CIFAR-10, MNIST	Custom, some ResNet	Custom, ResNet-18
(Jindal et al., 2024)	Caltech256, CIFAR-10, CIFAR-100, CUB-200-2011	ResNet-34	AlexNet, DenseNet-121, EfficientNet-B2, MobileNet-V3, ResNet-34
(Beetham et al., 2023)	CIFAR-10, CIFAR-100, FMNIST, GTSRB, MNIST, SVHN	ResNet-34	ResNet-18
(Hong et al., 2023)	CIFAR-10, FMNIST, MNIST, SVHN	ResNet-34, VGG-16	VGG-11
(Yang et al., 2024)	CIFAR-10, GTSRB, VGG Flower	Custom	Custom
(Li et al., 2025)	CIFAR-10, FMNIST, GTSRB, SVHN	some MobileNet, some ResNet, some VGG	some MobileNet, some ResNet, some VGG

Equations

$$Accuracy=\frac{1}{N}\sum _{i=1}^N\text{mathbbm}1(\hat{f}(x_i)=y_i).$$

$$Fidelity=\frac{1}{N}\sum _{i=1}^N\text{mathbbm}1(\hat{f}(x_i)=f(x_i)).$$

$$Transferability=\frac{1}{N}\sum _{i=1}^N\text{mathbbm}1(\hat{f}(x_i^{\prime })\ne \hat{f}(x_i)⟹f(x_i^{\prime })\ne f(x_i^{\prime })),$$

Full text

Model stealing, machine learning security

††copyright: acmlicensed††journalyear: 2025††doi: XXXXXXX.XXXXXXX††ccs: General and reference Design††ccs: General and reference Evaluation††ccs: Security and privacy††ccs: Security and privacy Software reverse engineering††ccs: General and reference Computing standards, RFCs and guidelines

1. Introduction

With the rising popularity of machine learning and its increasing deployment within real-world, practical applications and its ensuing exposure to end-users, the security aspects of machine-learning-based systems have gained wider attention. Several works showed that all machine learning models are vulnerable to attacks on their confidentiality, integrity, and availability, e.g., by evasion attacks in the form of adversarial examples (Szegedy et al., 2014; Biggio and Roli, 2018).

In recent years, aspects of the intellectual property (IP) of machine learning models and systems exploiting those models have attracted more attention (Oliynyk et al., 2023) and are a concern of practitioners (Grosse et al., 2023). Especially relevant here is that under current legislation, machine learning models are protected as trade secrets (Foss-Solbrekk, 2021). In other words, if they are leaked, their ownership cannot be protected by legal means. At the same time, machine learning models shared as black-box APIs are vulnerable to attacks at inference time, like model stealing (model extraction) attacks (Tramèr et al., 2016). While attacks have been investigated for different stealing goals against a vast range of machine learning tasks (Oliynyk et al., 2023), ensuring comparability of attacks following the same objective became more crucial. Currently, the largest attack group, consisting of more than 40 works, is focused on stealing the behaviour of image classifiers through training a substitute model with similar functionality. However, the exact attack carried out can differ significantly, depending on the attacker’s knowledge and capabilities – which in turn leads to different levels of attack impact and difficulty. Consequently, a fair attack comparison becomes more challenging, as comparing only a quantitative performance measure of attacks is not enough. Moreover, as different metrics have been proposed to quantify the success rate of model stealing attacks (e.g. (Jagielski et al., 2020)), there is no unified way of performance reporting, and many works in the field report only an insufficient subset of relevant measures. This inability to objectively evaluate an attack leads to ambiguity in defining the current state-of-the-art and slows down both the research on attacks and, more crucially, countermeasure development.

Benchmarking and evaluation are important aspects in many related fields, such as information retrieval (Armstrong et al., 2009), machine learning for classification problems (Fernández-Delgado et al., 2014), fairness in computer vision (Gustafson et al., 2023), or robustness of machine learning models (Rauber et al., 2017; Croce et al., 2021). In the area of model stealing, we lack well-established practices for attack evaluation, and more crucially, we lack an understanding of which current practices are wrong or insufficient.

To the best of our knowledge, the evaluation methodology of model stealing attacks has not been addressed before. With this paper, we therefore want to start a discussion about the criteria and recommendations that should be followed to ensure a fair, comprehensive, and comparable evaluation of model stealing attacks. For that purpose, we analyse substitute training attacks on image classifiers, which is the most actively developed sub-area of model stealing attacks.

Based on our analysis, we provide a set of recommendations for future work in the field to be considered when evaluating and comparing new approaches. Our main contributions are:

• •

A comprehensive threat model on possible levels of attackers’ knowledge, capabilities, and goals

• •

A systematisation of works on stealing image classification models along the proposed threat model

• •

The first framework for comparison of model stealing attacks that is based on the threat model

• •

An outlook of the current research status of the field based on the proposed framework, clearly demonstrating that (i) only a small fraction of prior attacks can be compared with each other, and (ii) several attack configuration have not been studied at all and a quarter of configurations have only been studied in one or two works

• •

An exhaustive analysis of previous works’ experiment setups in terms of learning tasks, model architectures, and queries that define conditions and limitations of the performed evaluation

• •

Best practices that incorporate recommendations for implementation before, during, and beyond experiments to ensure transparent and comprehensive attack evaluation

• •

An extensive list of current open questions related to the design and evaluation methodology of model stealing attacks

The remainder of this paper is organised as follows. Section 2 introduces terminology relevant for this work and discusses two similar areas, knowledge distillation and black-box adversarial attacks. Subsequently, in Section 3, we present a comprehensive threat model for model stealing attacks on image classifiers, and present challenges we encountered while deriving it. Further, we address comparability of attacks in Section 4 by introducing our framework for selecting compatible prior attacks. In Section 5, we analyse frequent attack setups in terms of classification tasks, model architectures, and query patterns. We then distil all prior observations into best practices in Section 6 and open research questions in Section 7. We further discuss how our findings transfer to other problem domains in Section 8. Finally, Section 9 gives an overview of related work in the field of evaluation methodology and Section 10 concludes the paper.

2. Background

We begin with an overview of model stealing attacks, introducing the necessary terminology and notation. Further, we discuss two concepts that, while having different goals than model stealing, in some scenarios employ similar techniques – knowledge distillation and black-box adversarial example attacks.

2.1. Model Stealing

Model stealing is an attack on the confidentiality of a machine learning model that aims at (partially) cloning the model by approximating its behaviour or obtaining information about its architecture, training hyperparameters, or learnable parameters (weights). In this work, we focus on attacks that aim to approximate the behaviour of image classification models.

The model that an attacker tries to steal is called the target model. We assume that the target model is exposed to third parties as-a-service, so that clients can send input queries to the model and obtain corresponding outputs (predictions). Depending on the service functionality, model predictions can differ in granularity and expressiveness. For a target model $f$ that aims to classify every input query $x$ as one of $k$ classes $\{c_{1},\ldots,c_{k}\}$, in this paper, we differentiate between three types of outputs. The first type is the prediction label $y$ (also called hard label), which only displays the class $y=c\in\{c_{1},\ldots,c_{k}\}$ to which query $x$ belongs according to the target model $f$, i.e. $f(x)=y$. The second type is prediction probabilities $(p_{1},\ldots,p_{k})$ (also confidence scores, soft labels or logits) that for each class $c_{i}$ show the probability $p_{i}$ of query $x$ belonging to class $c_{i}$. The third type of output is gradient-based explanations $e$, which are often returned in addition to the label $y$ or probabilities $(p_{1},\ldots,p_{k})$. Such explanations, for instance, Grad-CAM (Selvaraju et al., 2020), exploit gradients computed with respect to a particular layer (typically the last one) to display the most influential regions of the input image.

To approximate the behaviour of the target model, the adversary can use the target model as an oracle to collect predictions for the attacker’s data $X$, which consists of individual queries $x_{i}\in X$. Depending on the prediction types defined above, the adversary might obtain labels $y_{i}\in Y$, probability scores $(p_{1},\ldots,p_{k})_{i}\in P$, or gradient-based explanations $e_{i}\in E$, typically returned together with labels or probabilities. By combining the attacker’s data $X$ with corresponding predictions $Y$ (or $P$, $Y\cup E$, $P\cup E$), the adversary can create their own dataset and train a so-called substitute model $\hat{f}$ that aims to replicate the behaviour of the target model $f$.

The size of the attacker’s dataset $|X|$, i.e., the number of queries the adversary sends to the target model, is a common measurement for assessing the efficiency of an attack. For assessing the effectiveness, three metrics are commonly used: accuracy, fidelity, and transferability. Accuracy shows the performance of the substitute model on the original learning task of the target model, for instance, by a test set $\{(x_{i},y_{i})\}_{i=1}^{N}$, i.e.

[TABLE]

Fidelity shows similarity between the substitute and target model predictions:

[TABLE]

Finally, transferability shows how many of the adversarial examples crafted to fool the substitute model can also fool the target model into predicting the wrong class:

[TABLE]

where $x_{i}^{\prime}=x_{i}+\varepsilon$ is an adversarial example for the substitute model $\hat{f}$, i.e. $\hat{f}(x_{i}^{\prime})\neq\hat{f(x_{i})}$ for a small $\varepsilon$.

While having only queries as an interaction channel with the target model is a common assumption for stealing models offered as a service, there is a whole group of model stealing attacks that assume side-channel access to the target model (Oliynyk et al., 2023). This includes hardware, e.g. electromagnetic (Batina et al., 2019) or power (Xiang et al., 2020) side-channels, and software side channels like time (Duddu et al., 2019) or cache (Yan et al., 2020). Side-channel attacks exploit leaked information to reveal the exact model properties, such as architecture or learned parameters. On account of (i) the stealing goal we consider (behaviour approximation) being different, and (ii) side-channel attacks assuming different attacker’s capabilities that are not straightforwardly comparable with the query-based attacks, we exclude side-channel attacks from the scope of this work.

The diversity of data domains and tasks that are targeted with model stealing attacks keeps increasing. While analysing the field, we gathered 97 papers introducing methods for stealing the behaviour of machine learning models, almost half (47) of which focus on image classifiers, thus forming the most significant group of papers. Among other image-related tasks, attacks target encoders (Liu et al., 2022a), and object detectors (Liang et al., 2022). In the text domain, attacks have evolved from stealing simple recurrent neural networks for text classification (Pal et al., 2019) to more complex BERT models (Krishna et al., 2020) and even large language models (Gudibande et al., 2024). Graph data was also targeted in several works targeting graph neural networks (DeFazio and Ramesh, 2020; He et al., 2021a). Besides, model stealing attacks also appeared in fields of recommender (Yue et al., 2021) and autonomous driving systems (Zhang et al., 2022b). However, studies outside of image classification are rather scattered at the moment and have not reached a sufficient level of development to draw overarching conclusions regarding the evaluation methodology.

2.2. Knowledge Distillation

In the following, we describe a concept similar to model stealing – knowledge distillation. In particular, we discuss different categories of methods and which of them can be adapted to model stealing.

Knowledge distillation (KD) is a model compression technique that aims to distil knowledge from a large deep neural network (the teacher) into a simpler, smaller network (the student) (Hinton et al., 2015). Although white-box access to a teacher and availability of original training data are commonly assumed for KD, various scenarios similar to model stealing have been proposed. A recent survey by Gou et al. (Gou et al., 2021) classifies KD approaches based on (i) the knowledge a student obtains from the teacher, (ii) the distillation scheme, (iii) the teacher-student architecture, and (iv) the algorithm. Following their categorisation, KD techniques can be utilised for model stealing through substitute model training if they (i) are response-based, i.e., the student only obtains outputs of the teacher, and (ii) use offline distillation, i.e., the student is trained after the teacher. The similarity between KD and model stealing was also spotted by Ma et al. (Ma et al., 2021) and later utilised by two works to perform stealing attacks (Kundu et al., 2021; Jandial et al., 2022). Although some papers have been published on response-based offline knowledge distillation (also called black-box KD) (Wang, 2021) and data-free knowledge distillation (Wang et al., 2024), exploring the suitability of KD methods for model stealing is out of the scope of this paper and is left for future work.

2.3. Black-box Adversarial Example Attacks

Another field that shares some similarities with model stealing is adversarial examples crafted under black-box access to the target model. Adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015) are minute perturbations to data samples that cause a model to change its prediction, e.g., to misclassify a sample to a different class. These attacks are primarily studied for the image domain but also apply to other data types. The aim of the perturbation is to be not (easily) human detectable while fooling the model with a high success chance. Numerous methods exist for crafting these examples, mainly requiring white-box access to the model to be attacked.

There are two groups of attacks that operate under black-box access. The first group trains a substitute model to simulate the decision boundary of the target model and relies on the transferability of adversarial examples (Papernot et al., 2017). The second group directly optimises adversarial examples by approximating gradients of the target model (Chen et al., 2017). Below, we elaborate on mechanisms exploited by the first group, which has a similar methodology to model stealing.

Papernot et al. (Papernot et al., 2016a) were the first to demonstrate that one can employ model stealing to create a substitute model, and then craft adversarial examples for that stolen model with any white-box technique. If the two models are similar, adversarial examples crafted for the substitute model will also have a similar probability of succeeding in fooling the target model. Several further works focused on specifically optimising the training process for obtaining high transferability scores (Zhou et al., 2020; Chen et al., 2024; Sun et al., 2022).

While such black-box adversarial attacks have similar approaches to model stealing, they often can not be easily compared to approaches focused on model stealing itself, as they differ in purpose and evaluation. Many works only measure transferability, but not the other properties generally required for a successful model theft, like the performance of the stolen model on the original task of the target model. Therefore, we do not consider works that only address transferability in our analysis. However, we do recommend evaluating the fidelity and accuracy of the substitute models trained with these methods, as they might constitute significant progress over current methods.

3. Threat Model Overview

We start by analysing the threat models covered in $47$ relevant papers in Table 1. The threat models reported in that table are primarily derived by us from the experimental setup of the papers, as (i) only a minor fraction of the papers explicitly defined their threat models, and (ii) in some cases, the defined threat model was not followed and consequently verified in any of the experiments. While some of the approaches can potentially be applied under other conditions, one would need to verify this assumption with an empirical evaluation.

Based on the aspects of the threat model described in (Biggio and Roli, 2018), in the following, we characterise the attacker’s knowledge, capabilities, and goals for substitute training attacks on image classifiers and how they are represented in Table 1.

3.1. Attacker’s Knowledge

The attacker’s knowledge majorly defines the strength of an attack, consequently also characterising its difficulty. For model stealing, we identify three assets the attacker may have different knowledge of, namely the target model’s training data, the target model’s outputs, and the target model’s architecture. Further, we argue that there is another component that was overlooked in previous work – the usage of pre-trained models.

Target model’s data. In order to train a substitute model, an attacker has to create their own labelled dataset. This dataset usually starts with unlabelled data samples, which need to be labelled by the target model. However, the attacker’s dataset can significantly differ in quality depending on the attacker’s knowledge of the original classification task. We grouped prior works in Table 1 by the type of data available to the adversary: (i) original data that comes from the same distribution as the training data of the target model, (ii) problem-domain (PD) data that has the same context as the original data but a different distribution, (iii) non-problem-domain data (nPD) that does not fit into the context or distribution of the original data, and (iv) a data-free case, which assumes that no real data was used to obtain the substitute model.

(✢/✻/❋) Outputs of the target model. Predictions of the target model can be disclosed to end-users with different degrees of detail. The column ”Outputs” in Table 1 shows the outputs utilised by prior works, namely (i) (hard) labels (✢), (ii) confidence scores or logits (✻), (iii) both label-only and confidence score outputs were studied (✢/✻), (iv) explanations of predictions or gradients (❋), (iv) the outputs were not specified explicitly (-).

($\blacksquare$/$\square$) Target model’s architecture. If an attacker knows the architecture of the target model, they might utilise the same architecture to train a substitute model. In Table 1, we report this information under the column ”Model” with four possible options: (i) the substitute model has the same architecture as the target ($\square$), (ii) a different one ($\blacksquare$), (iii) both previous options were studied ($\blacksquare$/$\square$), and (iv) not enough information was provided to conclude any of the above (-).

Pre-trained models. Another aspect of the attacker’s knowledge concerns pre-trained models. The distribution of data used to pre-train weights of a model might have similar properties to the distribution of data from the target model’s learning task. Then, by using this model as a starting point for training a substitute, an adversary can gain an advantage from the distribution similarity and achieve better performance than with a substitute trained from scratch (Atli et al., 2020; Zhang et al., 2021). For image classification, for instance, a common practice is to use models pre-trained on ImageNet as a starting point for training substitute models.

Pre-trained auxiliary models like image-generating models or autoencoders can also be utilised to improve the performance of the attack. In particular, they can be used to initialise synthetic data generators for attacks with weak knowledge about the target model’s original training data (Barbalau et al., 2020; Lin et al., 2023; Yang et al., 2023a). As these models are usually task-specific, one can not assume that they are equally available or useful for all classification tasks. Moreover, if we go beyond the scope of image classification, the availability of public pre-trained models differs depending on the data domain. Therefore, the availability of pre-trained models should be considered as one of the factors that impact the strength of an attack. The discussion of whether the availability of pre-trained models should be considered part of the attacker’s knowledge was not addressed in previous works. Moreover, information about whether a substitute model is pre-trained is often omitted (see Section 5.3 for more details), which makes the analysis of prior works impossible. Hence, we omit this information from Table 1, but we encourage researchers to consider this in future evaluations.

3.2. Attacker’s Capabilities

It is a common assumption for substitute model training attacks that the target model is shared in a black-box manner (e.g., via an API) and that the only interaction possible is querying. Therefore, the query budget for conducting an attack is the main characteristic of the attacker’s capabilities.

( ) Query budget. We differentiate attacks on their query budget (QB) in Table 1, defined as the maximum number of queries spent in the corresponding paper. We split query budgets into four groups: (i) below 10,000 queries ( ), (ii) between 10,000 and 100,000 queries ( ), (iii) between 100,000 and 1,000,000 queries ( ), and (iv) more than 1,000,0000 queries ( ). In some works, information about queries was not provided (-).

3.3. Attacker’s Goal

We consider model stealing attacks that aim to train a substitute model with a behaviour close to the target model’s behaviour. Reaching this goal can be defined and quantified in two ways, accordingly to prior ways (Jagielski et al., 2020; Oliynyk et al., 2023). In the following, we discuss both of them and explain how they are incorporated in this work.

Jagielski et al. differentiate between three goals: functionally equivalent extraction, fidelity extraction, and task accuracy extraction (Jagielski et al., 2020). Creating a functionally equivalent model means that the substitute model should give the same predictions as the target model on the whole input space. Fidelity extraction releases this requirement to a specific input data distribution, usually, the one that corresponds to the task the target model was designed to solve. Both goals are measured using the same metric, fidelity. Finally, task accuracy extraction means that the substitute model has high performance on the original target model’s task, which is commonly measured using accuracy.

Another goal categorisation for behaviour stealing is proposed by Oliynyk et al. (Oliynyk et al., 2023). According to the authors, the substitute model aims to reach either the same level of effectiveness as the target model or prediction consistency with the target model. In the first case, the performance of the substitute is measured by accuracy. For prediction consistency, the prioritised metric depends on the data used for the evaluation. For real data, fidelity is measured, whereas for adversarial examples, it is transferability.

(AFT) Metrics. We combine the suggestions from both prior works (Jagielski et al., 2020; Oliynyk et al., 2023) and consider accuracy (A), fidelity (F), and transferability (T) as indicators of the specific goals set in the reviewed papers in the last column of Table 1.

3.4. Limitations and Challenges

To the best of our knowledge, we are the first to attempt a comprehensive analysis of model stealing threat models. Due to the lack of rigorous definitions, we encountered several challenges while classifying earlier attacks in Table 1. In the following, we express these challenges in the form of open questions, outline our suggested solutions, and describe their limitations. The way we resolve raised questions can only be taken as suggestions, as there are also reasonable arguments for other solutions. Primarily, we highlight those issues to bring more awareness to the lack of unified definitions and terminology.

3.4.1. Attacker’s Knowledge

While deriving assumptions about the attacker’s knowledge from prior work, we encountered difficulties in defining the types of the attacker’s data. Identifying if substitute and target architectures are the same or how detailed the outputs of the target models are primarily requires reporting corresponding information. Similarly, knowledge about pre-trained models can be communicated by explicitly reporting information about the utilised models. However, the body of related work is full of ambiguities when it comes to defining the type of data used. Below we list the most common controversies and how we mapped each of them into the categories in Table 1.

Original or problem-domain data? There are two positions on how original data can be defined. The first position considers any samples drawn from the distribution of the target model’s training data as original data. The second position defines exclusively the training data of the target model as the original data. Therefore, if half of the training set of a dataset is used to train the target model, and the other half is used for the substitute model training, the data type can be classified as either original (the first position) or problem-domain (the second position). We agree that this scenario differs from training on exactly the same data, but compared to what we consider problem-domain data, the distribution of data used is significantly closer to the target model’s training data. Hence, if an adversary uses part of the original dataset on which the target model was trained (even samples unseen by the target model) to train the substitute model, we classify its data type as original in Table 1.

Problem-domain or non-problem-domain data? This question arises when a dataset that is supposed to be non-problem-domain data (nPD) has some contextual or sample intersection with the original data. We obtained two approaches to address this issue: (i) removing the same categories or samples from the attacker’s dataset, and (ii) reporting quantified intersection between classes or datasets, while keeping the data unmodified. While keeping this data gives a benefit to the adversary, such intersections are often unintentional and might also occur in real-world scenarios. As in all works more than 50% of data samples are indeed nPD, we classified all of them as ones using nPD data.

Data-free or non-problem-domain data? Another ambiguity is over the definition of a data-free attack. We encounter two points of view: (i) a data-free attack does not use any real data at any stage of a model stealing attack, and (ii) a data-free attack does not use any real data to train a substitute model. In the second scenario, nPD data may have been used, for instance, to train a synthetic data generator that will subsequently produce PD-like data for the substitute model training. In both cases, the substitute model itself is trained without any real data, i.e., in a data-free manner. However, if an attack relies on having some nPD data to train a generator, then its data availability assumption does not differ from an attack that directly queries nPD data and trains a substitute model on it. On the other side, attacks that do not query nPD data often rely on open-source pre-trained generative models, implicitly using nPD data. For this reason, we categorised as data-free all attacks that do not use real data for training a substitute, regardless of the data used to train the generative model.

3.4.2. Attacker’s Capabilities

We represented attacker’s capabilities in Table 1 as the maximum number of queries used by an attack. This results in two questions concerning queries: (i) which number we should actually consider, and (ii) whether counting queries is a representative metric for the attacker’s capabilities and the efficiency of the attack.

Which number should we consider? Our first point concerns two terms that are sometimes used interchangeably in the literature: query budget and the number of queries. In Table 1, we reported query budgets, which we defined as the maximal number of queries used in a work. However, these numbers do not characterise how many queries an attack actually needs; it is an upper bound that the authors defined for their work. In fact, an attack performance might converge using fewer queries. This difference between the upper bound and the actually required amount of queries represents the difference between the query budget and the number of queries. While using the number of queries would reflect the strength of the attack better, deriving these numbers from related works remains challenging. We elaborate more on this question in Section 7.3.

Is counting queries enough? The absolute number of queries does not consider the difficulty of stealing. Such difficulty can be represented by the complexity of the target model ((Oliynyk et al., 2023)) or by the complexity of its classification task. Therefore, a more explanatory metric would include the number of queries (or the query budget) in relation to the size of the target model or its training dataset. Another issue is that the number of queries (or query budget) allows comparison of attacks with each other, but is not a very interpretable metric for real-world scenarios. Instead, measuring the actual cost of training a substitute model would provide more insights into the attack efficiency. At the moment, it remains an open question how to actually estimate the price of the model. However, in terms of model stealing, one can estimate the cost of querying the target model by using available APIs, which can serve as a lower bound of the model cost.

3.4.3. Attacker’s Goal

Aligning prior works based on their goals is essential for their comparability. As we can see from Table 1, no metric was reported in all papers, making a comparison of some methods complicated. We dedicate the whole following to the problem of comparability, whereas below we focus on another issue – measuring transferability.

How to measure transferability? Transferability has no commonly agreed-upon method of measuring, unlike accuracy and fidelity. A major obstacle is that one needs to decide which adversarial example crafting technique to use and set its hyperparameters, which heavily influences the difficulty of achieving high transferability and – it is generally easier to achieve high transferability scores on adversarial examples that contain larger perturbations. This, in turn, may influence the scores even more than the actual stealing method. Whereas we report transferability as one of the goals in Table 1, the actual scores can thus not be used to compare the attacks. To illustrate this reasoning, we gathered information from 12 papers that reported transferability scores in Table 2. Whereas there are clearly two dominant approaches, namely the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) and Projected Gradient Descent (PGD) (Madry et al., 2018), the maximum perturbation value that regulates the strength of the perturbation varies from paper to paper, making the comparison unfair

4. Framework for Attack Comparison

In this section, we introduce our framework for attack comparison. The framework is built upon the threat model defined in the previous section. As the first step, we define how each individual attack can be evaluated in terms of effectiveness and efficiency. Then we discuss how to compare different attacks based on their threat models. To this end, we outline attack characteristics that should be consistent to ensure comparability and propose an attack categorisation based on them. Finally, we apply the proposed categorisation to prior work and use it to identify current limitations and challenges in the field.

4.1. Attack Evaluation

Before comparing different attacks, we first have to identify how each individual attack is evaluated. Below, we demonstrate how the attacker’s goal and capabilities defined earlier in Table 1 are naturally aligned with the effectiveness and efficiency of an attack.

Effectiveness. We characterised the goal of an attack in Section 3.3 by three metrics: accuracy, fidelity, and transferability. Each of these metrics represents the effectiveness of an attack. We discussed difficulties of using transferability to compare different prior attacks in Section 3.4.3. Whereas it is a valid metric for evaluating the similarity of target and substitute models near their decision boundaries, it requires further agreed-upon standards to be consistently measured and comparable across different works. However, both accuracy and fidelity are unequivocally defined in the literature and can be used to compare the effectiveness of attacks. The only constraint is that the evaluation has to be performed on the same test data.

Efficiency. We identified in Section 3.2 the capabilities of an attacker with the query budget of the attack. In Section 3.4.2, we discussed how the capabilities can be characterised by other values, for instance, the number of queries used per target model’s parameter. Nevertheless, all these metrics are based on the query count and, therefore, represent the efficiency of an attack.

The attacker’s goal and capabilities as defined in Section 3 explicitly correspond to the effectiveness and efficiency of an attack. Therefore, they enhance comparability by offering explicit numerical values for direct comparison. However, different assumptions about the attacker’s knowledge lead to different levels of attack difficulty. For a fair comparison, we should first ensure that attacks are being performed under the same restrictions that originate from the attacker’s knowledge. To this end, we propose a categorisation of attacks based on the attacker’s knowledge, which ensures that attacks from the same category are comparable.

4.2. Attack Categorisation

The second part of our framework is an attack categorisation, which ensures that attacks within the same category are comparable. By design, and as also suggested for other related fields, such as evasion attacks (Carlini et al., 2019), model stealing attacks should be examined within their threat model as defined in Section 3. Usually, attacks need an adaptation to be launched in a broader spectrum of settings. Table 1 demonstrated how vastly threat models can differ from paper to paper.

However, as was established in the previous section, the attacker’s capabilities and goals are aligned with the attack evaluation. Therefore, an attack approach can be easily adapted to become comparable with others by reporting accuracy, fidelity, transferability (effectiveness), and the number of queries (efficiency). In contrast, changing the attacker’s knowledge can significantly impact the methodology of an attack. For instance, providing real data to a data-free attack vanishes the primary purpose of an attack being data-free. Hence, comparing two model stealing attacks is, in principle, only meaningful if these attacks assume the same attacker’s knowledge. To this end, we devised a graphical representation of the attacker’s knowledge and grouped all analysed papers based on this assumed attacker’s knowledge, shown in Figure 1.

The diagram is constructed as follows. The dashed vertical line divides the diagram into two parts: the left half corresponds to settings where the target and substitute architectures are different, and the right half corresponds to settings that use the same architecture for both models. Sectors spanning the concentric circles correspond to particular knowledge about the data: the data-free assumption is on the top, followed by non-problem-domain (nPD) and problem-domain (PD) data, and original data at the bottom. Each sector is divided into segments by three concentric circles, which represent knowledge about the target model outputs. The inner circle corresponds to labels, the middle circle to probabilities, and the outer circle to explanations and gradients. Through this partitioning, each derived segment corresponds to a specific attacker’s knowledge, covering altogether $24$ different scenarios.

We map knowledge assumed by each paper from Table 1 into a segment corresponding to the same knowledge by placing a reference to the paper into the segment. If a paper covers different scenarios that fit into several segments, we put the corresponding reference in each segment. If some information about the knowledge is not given, we place its reference into each potential segment and highlight it in grey. We fill empty segments in dark grey, meaning that there are currently no works that consider the attacker’s knowledge associated with those segments.

4.3. Trends and Observations

Figure 1 demonstrates the current coverage of different categories in the field of attacks against image classifiers. We identify the following trends:

(i) For a fair comparison, one should only consider attacks within the same segment, meaning that only a small fraction of works are actually comparable to each other. The largest segment (which represents different architectures, nPD data, and only labels as output) contains at most 11 papers (2 works have undefined knowledge about the target architecture), which is less than 25% of the analysed papers.

(ii) A quarter of the segments have no prior work at all (the grey segments), a third have at most one publication, and more than half have at most three prior publications. At the same time, three segments have nearly 10 works, demonstrating a strong imbalance among threat models investigated in prior studies.

(iii) More than half of the segments with explanations as target output knowledge are not explored at all. In particular, none of the papers explore how to launch a substitute training attack having explanations (or gradients) and either PD or nPD data.

(iv) The diagram is imbalanced towards the upper left, inner part, indicating that more papers tend to assume a weaker attacker’s knowledge. A well-performing attack with a weaker knowledge assumption raises more threats than one with a stronger assumption, as it can be successfully launched against a ”more protected” API, i.e., one that does not disclose a lot of information about the target model and the original data. However, attacks with a stronger assumption are more useful for estimating the worst-case scenario when testing defences. Designing new attacks in the lower part of the diagram is, therefore, more beneficial for defence development.

(v) Complementing the previous point, we notice that the area of attacks that assume the availability of PD data is the least developed among data-related assumptions of the attacker’s knowledge. However, especially for image classification, PD data can actually be the most practical scenario. The development of practical attacks facilitates a better understanding of which defences should be applied in real-world scenarios. Therefore, we would like to encourage the community to explore more practical scenarios.

5. Frequent Attack Setups from Related Work

In the previous sections, we analysed prior work in terms of threat modelling and comparability. In this section, we study a more practical side of established attacks. We provide insightful statistical information about the examined papers to demonstrate which experimental setups were the most frequent in the recent works. In particular, we examine which image classification tasks were addressed, which target and substitute models were used, and how many queries were utilised. The overview of setups used in each work is summarised in Appendix A, whereas, in this section, we provide statistics to examine the general comparability between different works.

5.1. Dataset

The classification task of the target model is mainly defined by its training dataset. We thus report the commonly used target model datasets in Figure 2(a). If, within the same work, experiments were conducted for different original datasets, we counted the corresponding paper multiple times. Overall, more than 20 different datasets were used. The most popular dataset is CIFAR-10 (38 papers), followed by MNIST (21), Fashion MNIST (FMINST) (18), Street View House Numbers (SVHN) (15), German Traffic Sign Recognition Benchmark (GTSRB) (13), and CIFAR-100 (8). Additionally, we analysed how many datasets are used for training target models in each paper, with the results presented in Figure 2(b). Most papers study either two datasets (16 papers) or four (15), as visible from the two peaks.

5.2. Target Model

Another important factor besides the dataset is the target model architecture; statistics of usage of popular architectures are shown in Figure 3(a). As with datasets, if several target models were trained in one paper, they were counted multiple times. The most popular choices are ResNet34 (20), AlexNet (9), LeNet (9), ResNet18 (9), VGG16 (6), ResNet50 (6), and VGG19 (5). However, the information presented is not complete: in six papers, information about the target model was (at least partially) missing, and in 13 papers, custom CNN architectures were utilised.

Subsequently, we aligned the most popular dataset choices with the target architectures to examine the relationship between those two choices. Table 3 demonstrates how many times the most common target architectures were used to learn the most common datasets. Each paper can contribute multiple times, as we counted the total number of model-dataset combinations across all 47 works. As a result, some papers contributed significantly more entries than others. For example, Yan et al. (Yan et al., 2023c) trained 16 different target models for each of four target datasets (CIFAR-10, MNIST, FMNIST, CIFAR-100), contributing 64 entries to Table 3. For brevity, we only list the top 7 most common architectures. Yan et al. (Yan et al., 2023c) experimented with 5 of them, meaning that the remaining 11 architectures contributed to the last row of each of the four datasets in Table 3.

Only for the SVHN dataset, the majority of experiments were conducted using one of the listed architectures, namely ResNet34. However, if we deduct the 11 entries of Yan et al. (Yan et al., 2023c) from the last row (who use CIFAR-10, MNIST, FMNIST, and CIFAR-100), the ResNet34 model is also the most common option for CIFAR-10 and CIFAR-100. For MNIST and FMNIST, among the listed architectures, LeNet is the most typically used. For GTSRB, the most frequently used architecture is a custom CNN (see Appendix A for more details).

Finally, we examined the adoption of transfer learning for training target models. Only 7 out of 47 examined papers explicitly mention whether the target model was trained from scratch or from pre-trained weights. Only a single work reported training the target model from scratch, three reported using pre-trained weights, and three more works adopted both strategies.

5.3. Substitute Model

Subsequently, we investigated substitute model architectures with the results presented in Figure 3(b). Here, ResNet18 (22) is the most widely used architecture, followed by VGG16 (9), ResNet34 (7), LeNet (5), half-AlexNet (5), which is a customised version of AlexNet with half the capacity, ResNet50 (5), AlexNet (4), and VGG11 (4). As with the target model, 13 papers used custom CNN architectures. Four papers did not report the substitute architecture for at least some datasets. As with target architectures, we examined the usage of transfer learning for training substitute models. Whereas most works (29) do not explicitly mention from which weights the substitute model is trained, the number of works that report this information (18) is significantly larger than for target models (7). Among those 18 works, 9 train substitute models from scratch, 7 start with pre-trained weights, and two works investigate both approaches.

We present how often the most popular substitute architectures were used to steal the most popular target architectures in Table 4. As stated earlier, the contribution of different papers varies significantly from just a single entry to 64 entries (four substitute models against four target models trained for each of four datasets (Yan et al., 2023c)). The latter has significantly contributed to the ”Others/Others” setting (right bottom cell), as a significant part of the combinations contains architectures that are not that common. However, unlike with the datasets, for most of the target architectures, there is a match from the substitute list. ResNet18 was mostly used for stealing ResNet34 and ResNet18, LeNet and VGG16 were used to steal themselves, and for AlexNet the most common substitute choice was half-AlexNet. For VGG19 and ResNet50, Table 4 is inconclusive.

5.4. Number of Queries

The last attack setup component we analysed is the number of queries reported. We plot the frequency of using a certain number of queries in Figure 4, highlighting the datasets from Figure 2(a). The number of queries is binned into different groups for better visual comprehension. Most of the experiments were conducted with fewer than 50,000 queries, with especially many in the range between 10,000 and 50,000 queries. Barely any results were reported for two groups: between 50,000 and 100,000 queries and between 500,000 and 1,000,000 queries. For CIFAR-10 and SVHN, a lot of experiments were performed with over 1 million queries. Further investigation revealed that this trend is caused by data-free attacks. Unlike attacks that use real images to train a substitute model, attacks that rely on artificial data usually require millions of queries to converge (Hong et al., 2023; Yang et al., 2023b). In turn, attacks that utilise real images are constrained by the size of the dataset from which the images come.

While Figure 4 gives an overall picture, the number of queries is not representative enough for comparability purposes (see our discussion in Section 3.4.2). For this reason, we calculated the ratio between the number of queries and the target model’s training set size. This ratio shows how many queries an attacker spends per sample from the target model’s training data. We plot it against the number of queries in Figure 5.

Only for two datasets, MNIST and CIFAR-10, was a very low number of queries (below 100) studied. Most of the works, as stated above, used up to 50,000 queries, with 0.01 to 1 queries spent per target model’s training data sample. Those attacks, therefore, used at most as many samples as there are in the target model’s training dataset, making them relatively efficient. On the other side, data-free attacks with millions of queries spent correspond to at least 100 queries per training sample.

6. Best Practices

In this section, we summarise our observations from prior sections into guidelines for designing and evaluating model stealing attacks. We gather best practices from previous work that can be performed before, after, and beyond experiments and present them as a list of recommendations (Rs).

6.1. (R1) Before Experiments

We begin by discussing recommended strategies for developing and evaluating model stealing attacks before the experimental part starts. In particular, we emphasise the importance of a clear threat model and an adequate experimental setup.

(R1.1) Threat model. Well-defined threat models are essential for ensuring attack comparability and countermeasure development. In this work, we primarily focused on attacks that intend to copy the behaviour of the target model, for which we defined a detailed overview of the attacker’s profile in Section 3. For other attackers’ goals, we direct readers to a related survey (Oliynyk et al., 2023).

For behaviour-stealing attacks based on substitute model training, the following aspects of the attacker’s knowledge and capabilities must be clearly specified:

• •

Attacker’s data. As we defined in Section 3.1, an attack can use original, problem-domain, non-problem-domain data or no real data at all (i.e., data-free attack).

• •

Target model’s architecture. Knowing the target model’s architecture can significantly simplify the selection of the substitute model’s architecture: using exactly the same architecture for both models leads to higher transferability scores (Juuti et al., 2019), giving even more advantage to the adversary.

• •

Pre-trained models. As discussed in Section 7.2, pre-trained models can give a significant performance improvement to the adversary. Therefore, authors should specify whether the target, substitute, or any auxiliary model is pre-trained and should report which dataset was used.

• •

Target model’s outputs. For classification problems, we differentiate between labels, probabilities (confidence score), and gradients or explanations as outputs, as was defined in Section 3.1. Prior works have demonstrated that having outputs that reveal more information leads to better-performing attacks: probabilities give better results than labels (Orekondy et al., 2019; Juuti et al., 2019; Atli et al., 2020), and explanations render better models than probabilities (Miura et al., 2021).

• •

Query budget. All limitations and capabilities of the attacker in terms of query budget numbers must be explicitly stated. In addition, if an attack requires a validation set labelled by the target model, its size should ideally be included in the total query budget, as done by Jindal et al. (Jindal et al., 2024).

(R1.2) Experiment Setup. Defining an appropriate experimental setup is another crucial step before launching the experiments. In particular, it should correspond to the defined attacker’s knowledge and capabilities and be sufficient to benchmark the results against the state-of-the-art. The statistics reported in Section 5 can serve as a good starting point for defining attack setups.

• •

Target dataset. As we mentioned earlier in Section 7.1, large and complex datasets have not been widely studied before. Therefore, researchers should explore a variety of target datasets with different complexities. Besides, if the baselines defined in the previous step are earlier attacks, using the same target datasets is crucial for comparability.

• •

Attacker’s data. The attacker’s dataset must align with the defined threat model. In Section 7.1, we discussed that semantically, the same attacker’s knowledge about data could be interpreted differently. Therefore, following previous work (Orekondy et al., 2019), researchers must verify whether the attacker’s data has any (unintentional) intersections with the original target model’s training (or test) data. Furthermore, removing common data from the attacker’s set can potentially make the actual setup closer to the defined threat model. Additionally, authors must specify whether any additional data was used for training an auxiliary model (Lin et al., 2023; Yang et al., 2023b).

• •

Target model. Similarly to datasets, complex state-of-the-art models are under-represented in the current research, as was mentioned in Section 7.2. While exploring more complex models is beneficial for evaluating attack generalisability, simpler models might be needed for comparability with previous work (see Section 5.2). For a fair comparison, either exactly the same target model can be used (Rosenthal et al., 2023), or the target model can be trained until it achieves approximately the same performance as in previous work (Sanyal et al., 2022).

• •

Substitute model. The choice of substitute model should align with the assumed knowledge about the target model, i.e., if the target architecture is unknown, the substitute architecture should be different.

6.2. (R2) During Experiments

We continue the discussion with best practices that should be applied during model stealing experiments. At this stage, a proper evaluation of attacks is the most crucial. We approach it from several perspectives: defining appropriate baselines, performing exhaustive effectiveness and efficiency evaluation, and conducting ablation studies.

(R2.1) Baselines. Defining appropriate baselines is essential for aligning novel attacks with the state-of-the-art. We recommend using Table 1 and Figure 1 to identify state-of-the-art attacks with an identical threat model. Among others, these attacks are the most feasible and fair baselines for a novel attack.

(R2.2) Effectiveness evaluation. As was demonstrated in Table 1, none of the metrics (accuracy, fidelity, transferability) was reported for every previous attack. While accuracy is the most common, it is only meaningful when the accuracy of the target model is specified as well. Fidelity is more self-explanatory, but it is often missing from the attack assessment (Table 1). Therefore, to ensure a comprehensive effectiveness evaluation, we recommend including both metrics.

As discussed in Section 3.4.3 and illustrated in Table 2, there is no common practice on how transferability should be measured for model stealing attacks. Previous work disagrees on both methods for crafting adversarial examples and samples on which adversarial perturbations are applied. However, transferability is heavily investigated in the adversarial example community (Papernot et al., 2016a; Gu et al., 2023). In contrast to model stealing, research has brought forward benchmarks (Croce et al., 2021) and related strong attacks (Croce and Hein, 7 18) as well as research on failure modes of attacks (Pintor et al., 2022). Evaluations for model stealing should draw from this literature to obtain more reliable evaluations.

Finally, if the effectiveness evaluation is performed on a dataset with a predefined test set, the latter should remain unmodified. While the test set could be a good source of original data used for an attack, leaving only a fraction of it for actual testing may lead to some discrepancies. In general, this may not invalidate an attack, but it makes the comparison with previous attacks difficult, especially if the difference in performance is not significant.

(R2.3) Efficiency evaluation. The primary indicator of the attack efficiency is the number of queries used to train a substitute model. Several sources can be used to identify potentially relevant values: (i) the pre-defined query budget (as discussed in Section 7.3), (ii) convergence of the substitute model (Hong et al., 2023), (iii) number of queries reported by baseline state-of-the-art attacks (Truong et al., 2021), and (iv) number of queries required to achieve the same performance as the baselines (Rosenthal et al., 2023). However, as mentioned earlier in Section 7.3, placing the number of queries in context with the complexity of the target model (Oliynyk et al., 2023) or its learning task (as was done in Section 5.4) gives a more objective efficiency assessment.

(R2.4) Ablation studies. Ablation studies are essential for understanding the contribution of each attack component and verifying whether attacks that incorporate additional techniques for performance improvement indeed perform better. From a certain perspective, they also produce baselines in the form of a basic attack on top of which the improvements were built. Therefore, authors should perform an ablation study for every (new) attack parameter. Good examples include (i) comparing an attack that uses two data types with an attack that only uses one type of data (Correia-Silva et al., 2018), (ii) evaluating how query optimisation techniques improve the attack performance compared to the non-optimised attack (Jagielski et al., 2020; Pal et al., 2020), and (iii) comparing the performance of an ensemble of substitute models with a single substitute model (Jindal et al., 2024).

6.3. (R3) Beyond Experiments

Finally, we outline additional best practices that go beyond standard experiments. They include validating attacks in practical scenarios, testing their ability to evade state-of-the-art defences, reporting negative results, ensuring the reproducibility, and disclosing vulnerabilities in an ethical way.

(R3.1) Real-world APIs. Real-world applications are commonly not a feasible setting for an adequate attack evaluation. Stealing a model, even for scientific purposes, may destroy the intellectual property of the owner of this model (Foss-Solbrekk, 2021), raising ethical issues. In addition, APIs usually operate on a black-box basis, hiding information about the (dynamically changing) target model and its learning task. Furthermore, deployed models often have constraints on the allowed queries (Grosse et al., 2024), potentially rendering attacks with huge query budgets infeasible. These restrictions can narrow down or even modify the initial threat model of an attack, leading to a less comprehensive evaluation. Moreover, some metrics might be impossible to measure, for instance, the accuracy of the substitute model on the original learning task or the efficiency score (the number of queries divided by the number of weights in a model). However, these conditions correspond to a real attack scenario, and understanding how dangerous attacks are against real-world APIs and to what extent they can actually be measured is vital for the development of countermeasures. Therefore, following previous works (Correia-Silva et al., 2018; Yu et al., 2020; Gong et al., 2021), we highly recommend evaluating novel approaches against real-world applications, while ensuring solid, i.e., non-evolving, setups that ensure interoperability.

(R3.2) Defences. Another important step in further attack evaluation is testing them against the state-of-the-art defence mechanisms. To ensure fair evaluation, it is crucial to select defences specifically designed for a concrete threat model. We refer readers to the recent survey for further information about defence categories and the line-up between attacks and defences (Oliynyk et al., 2023). In addition, general guidelines on how to evaluate defences should be developed, analogously to evasion attacks (Carlini et al., 2019), where testing against adaptive attacks is a de facto standard (Carlini et al., 2019).

(R3.3) Negative results. Additionally, we would like to encourage the community to share negative results. While often not as groundbreaking as positive results, they might provide key insights for future work. This is especially important for results related to countermeasure testing, as robust defence mechanisms are the common ultimate goal of the community. Furthermore, reporting negative results safeguards others from re-investing in the same research, saving computational and time costs. In turn, publications of negative results should be encouraged by the program chairs of publication venues.

(R3.4) Reproducibility. Reproducibility ensures comparability and enables future follow-up work. To this end, design and implementation processes need to be reported in detail. In particular, all components mentioned above, such as threat model, experiment setup, and evaluation, must be provided, either with the code or in the research output. The implementation should be publicly available with explicit documentation and instructions on how to recreate the experiment results.

(R3.5) Ethics. At the same time, releasing code may make it easier for malicious entities to perform an attack. Along these lines, it is of general importance to comply with best practices of disclosure111https://cheatsheetseries.owasp.org/cheatsheets/Vulnerability_Disclosure_Cheat_Sheet.html when a model is found to be particularly vulnerable to be stolen. Calls to implement the corresponding infrastructure for both vulnerabilities (Longpre et al., 2024) and incidents (Bieringer et al., 2025) have been made in the scientific community.

7. Open Research Questions

In this section, we discuss the open research questions (RQs) that emerge from our analysis. We organise them by first reviewing open questions relating to datasets, models, and then the number of queries.

7.1. (RQ1) Datasets

We identify several open research questions associated with the datasets used. First, we discuss open questions related to the used datasets and attack generalisation across datasets. Afterwards, we demonstrate the current lack of a formal definition of attacker’s knowledge about data.

(RQ1.1) Generalisation across datasets. We start with a discussion about the used datasets and attack generalisation across these. The statistics presented in Section 5.1 show that datasets like CIFAR-10, MNIST, and Fashion MNIST are most commonly studied. Yet, attacks launched against MNIST classifiers perform in general better than against more complex datasets, for instance, GTSRB (Juuti et al., 2019; Pal et al., 2020), SVHN (Yuan et al., 2022; Khaled et al., 2022) or CIFAR-10 (Yang et al., 2023b). A noise-based attack by Milli et al. (Milli et al., 2019) only works on MNIST classifiers, and is ineffective against CIFAR-10 classifiers, suggesting that attacks do not necessarily generalise. These findings underline the need to study more challenging tasks, such as CIFAR-100, which is currently only studied in 8 papers, being the 6th most frequent target dataset, compared to 45 (out of 47 examined) papers studying (Fashion) MNIST or CIFAR-10.

(RQ1.2) Attacker’s knowledge about data. Further open questions related to dataset complexity occur due to vaguely defined attackers’ knowledge about the training data. For example, Truong et al. (Truong et al., 2021) demonstrated that the similarity of attacker’s and original data domains significantly contributes to the attack effectiveness. On the other hand attacker data types are not clearly defined as we derived in Section 3.4.1: (i) current works have different understandings on what the original data consist of (the exact training data of the target model or the distribution this data was drawn from), (ii) data that at first sight looks like non-problem domain data can turn out to be problem-domain due to unintentional intersections between data domains, and (iii) in literature, a data-free attack stands for both an attack that does not use any real data to query target model and an attack that does not use any real data to train a substitute model, whereas querying with real data is possible to e.g. train an auxiliary generative model. In all these examples, semantically the same knowledge about the data can give more or less benefits for an attack. For a fair comparison, this should not be the case. Consequently, more research and, ultimately, standards are required to better distinguish differences in attacker’s knowledge types and understand their impact on attack success.

(RQ1.3) Overlap across datasets. Finally, it remains also an open question to which degree data is shared or accessible in practice (Grosse et al., 2023). We have loosely discussed this point when discussing the data originally used to train the models (cf. Sections 5.2 and 5.3). In real world applications, it remains an open question which data preprocessing steps have been carried (Grosse et al., 2024, 2023) out and how they affect potential dataset overlap when the same data is used but slightly changed due to pre-processing. An addition open question remains whether data was sourced from public sources and is thus accessible indirectly to the attacker (Grosse et al., 2024). Overlap in the data used should thus be object of study and documented in detail.

7.2. (RQ2) Models

We further present open questions related to various models that play a role in model stealing attacks, namely target models, substitute models, and auxiliary models such as generators. We start with discussing current limitations in attacks against complex, state-of-the-art models. Then, we outline open questions regarding the usage of pre-trained substitute and auxiliary models. Finally, we review partial stealing of models and whether it is a simpler and, therefore, more dangerous task.

(RQ2.1) Large and complex models. We begin with a discussion about attacks against complex state-of-the-art models. In Section 5.2, we presented the most common target architectures (with a comprehensive overview in Appendix A). However, none of these works studies more advanced architectures such as EfficientNet (Tan and Le, 2019), BigTransfer (BiT) (Kolesnikov et al., 2020), or vision transformers (ViT) (Dosovitskiy et al., 2021), which currently lead in CIFAR-10 classification task222https://paperswithcode.com/sota/image-classification-on-cifar-10. These architectures have not been studied neither in the context of other model stealing attacks, nor for training a substitute model (Oliynyk et al., 2023). Further research is needed to determine whether current attacks scale to more advanced and complex architectures.

This is especially relevant as recent work suggests that attacks might not easily generalise to complex models. Gudibande et al. (Gudibande et al., 2024) demonstrated that imitation of ChatGPT through fine-tuning open-source large language models is not successful. Although fine-tuned models learned to mimic the style of ChatGPT, with further evaluation, it turned out that there is a significant gap in the factuality of ChatGPT and fine-tuned models.

(RQ2.2) Pre-trained models. Another key question concerns the usage of pre-trained models by adversaries. Zhang et al. (Zhang et al., 2021) demonstrated that the availability of pre-trained substitute models could give a significant advantage to an adversary compared to models trained from scratch. Such an overlap is for example created when the target and substitute models were both pre-trained on the same data, which is also a subset of the training data of the target model. This concern has practical relevance, as there is evidence that pre-trained models are heavily used when deploying AI in industry (Grosse et al., 2024). However, as we mentioned earlier in Section 5.2 and Section 5.3, information on whether pre-trained weights were used for target or substitute models is often omitted.

Pre-trained models can also be implicitly used in the attack, for instance, to initialise a synthetic data generator (Barbalau et al., 2020; Yang et al., 2023b; Lin et al., 2023). In this case, an attack implicitly relies on the knowledge that the generator absorbed from its pre-training data. Even if only generated synthetic data is further used to train a substitute model, there is a significant hidden contribution of that preliminary generator’s knowledge. Therefore, the usage of pre-trained models needs to be explicitly communicated and potentially included in the attacker’s knowledge and capabilities.

(RQ2.3) Partial model stealing. Finally, we turn our attention to the problem of partial stealing. In real-world scenarios, an adversary might be interested only in a specific behaviour of the target model. In this case, the threat model defined in Section 3 needs to be modified, as (i) not all samples from the original data distribution are relevant, (ii) the substitute architecture needs to be modified accordingly to the goal task, and (iii) evaluation should be performed on task-relevant data. While very limited research has been done in this direction, current findings suggest that partial stealing can be easier and, therefore, potentially more dangerous than obtaining the full functionality. Okada et al. (Okada et al., 2020) considered a special scenario of model stealing attacks against image classifiers, in which a substitute model is trained to replicate the target’s behaviour only on a specific subset of classes. The results demonstrated that within the same query budget, the substitute model achieves higher scores when duplicating only partial functionality compared to full functionality (Okada et al., 2020). This finding suggests that partial stealing is effective while being more efficient than full functionality stealing. It remains an open question if partial stealing is always a threat and if it can be countered with the same methods that are applicable against complete functionality stealing.

7.3. (RQ3) Queries

The last group of open questions concern the attacker’s queries. We begin with a discussion around two similar terms: query budget and number of queries, which we will define below. Next, we analyse how practical the exact values from related work are and raise the question of their scalability. Finally, we outline the limitations of counting queries for attack efficiency assessment.

(RQ3.1) Number of queries vs query budget. As a starting point, we pose a question regarding understanding what query budget and number of queries characterise. While sometimes these terms are used interchangeably, we see a clear distinction between them. The number of queries is measured by an adversary as the query count required to perform their attack; it is defined by internal attack-specific constraints, e.g. how many queries are needed for the substitute model to converge. The query budget is, in contrast, a rather external constraint of how many queries an adversary can potentially make. There can be different sources of such constraints: limitations of the API, limited financial abilities of an adversary, monitoring defences (Juuti et al., 2019) that will cut off the adversary after a certain time, etc. To take full advantage of an attack, the number of queries it requires should not exceed the query budget. However, as there is no clarity in the literature concerning this terminology, different works use different sources of constraints (external vs internal) to define how many queries to spend on their attack. Driven by different incentives, those attacks essentially correspond to different threat models. The comparability of such attacks remains an open question, as well as a clear distinction of what impacts the decision to spend that exact amount of queries.

(RQ3.2) Practical query numbers. The next open question concerns the practicality of a number of queries considered in the literature. For that, we need practical query budgets, i.e. external constraints on the query count. Query numbers on deployed AI models, according to a recent survey, could be as low as less than 100 or less than 1,000; where in most cases, models cannot be queried at all (Grosse et al., 2024). While the authors (Grosse et al., 2024) do report unlimited amounts of queries as the second most frequent, these numbers may be reduced by applying a monitoring defence that can cut off suspicious clients after as little as 100 queries (Juuti et al., 2019).

Despite such strict practical constraints, the numbers required by attacks can be significantly higher. As shown in Figure 4, only a small fraction of works relied on less than 1,000 queries. Most of the experiments were conducted with budgets between 10,000 and 50,000 queries. Finally, data-free attacks, as noticed in Section 5.4, usually require millions of queries, which likely renders them useless under practical limitations. Therefore, more studies with very limited attacker’s capabilities are needed to understand how dangerous model stealing attacks are in practice. On the other side, we need to expand our knowledge about the practical scenarios to better identify potential threats.

(RQ3.3) Query scalability. Another practical issue with model stealing attacks is their scalability. In Section 5.4, we demonstrated that, on average, data-free attacks need more than 100 queries per target model’s training sample to steal a model. We can rephrase this observation in the following way: on average, it takes at least 100 queries to compensate for the knowledge that the target model obtains from a single training sample. The training data, in this case, consists of low-resolution CIFAR-10 images. It is an open question if such data-free attacks would scale for more complex, high-resolution data. At the moment, especially taking into account the practical number of queries reported above, such data-free attacks do not appear to be a practical threat against complex tasks.

(RQ3.4) Other efficiency metrics. Lastly, we would like to raise the question of whether measuring the number of queries is sufficient for efficiency evaluation. One key factor that can be taken into account is the complexity of the target model. Previous work (Oliynyk et al., 2023) suggested calculating a so-called efficiency score that shows how many queries per learnable parameter of the target model an attack requires. Another factor that we considered in this work in Section 5.4 is the complexity of the target model’s task, which we quantified as the number of samples in the training set. Including both model and data complexity in efficiency assessment can be a further step here.

Another efficiency estimation can come from the actual cost spent to train a substitute model. Previous work that, in particular, studied attacks against public APIs reported the price they had to pay for the executed queries (Yu et al., 2020). While this price gives a good estimate of how efficient the attacker’s data-gathering process is, the costs of actually training a model and finding appropriate architecture and hyperparameters are not included. To the best of our knowledge, estimating the actual price of creating a model from scratch is still an open question. This leads to difficulties in estimating the actual attack cost and whether it is reasonable compared to the price of creating a target model.

8. Discussion on Generalisation

In this work, we analysed the largest and most developed group of attacks that clone the behaviour of target models offered as a service - attacks on image classification models. In the following, we discuss how our analysis, framework and recommendations transfer to other problem domains.

Threat model. In Section 3 we defined the threat model for attacks against image classification models in terms of attacker’s knowledge, capabilities, and goals. In the following, we discuss how each of this components can be adapted to other problem domains. The attacker’s knowledge consists of four components: target model’s data, its outputs, architecture, and availability of pre-trained models. Attacker’s data defines potential queries and is required for any other domain as well. Similarly to image classification, one can further define different strengths of attacker’s knowledge about the data depending on the similarity of available data to the target model’s original training data. However, different data types may affect the attacker by adding or removing constraints (Grosse and Alahi, 2024). As an example, tasks like trajectory prediction, representing the trajectory of a physical vehicle, are more constrained regarding attack perturbations than images with real-valued pixels (Grosse and Alahi, 2024). An exploration of such constraints is, due to the lack of corresponding approaches, left for future work.

Regarding the knowledge of the model’s output, the target model returns outputs regardless of the domain; the granularity of the outputs is however defined by the domain problem and might differ from the one for image classification. Extending, merging, or adapting the three classes provided in our work may thus be necessary, as, for example, pixel-wise tasks like segmentation by default leak more information than classification. Further, the notion of attacker’s knowledge about the target and pre-trained models transfers across different domains in a straightforward way. Attacker’s capabilities, defined by the number of queries, also transfer to any query-based attack regardless of the problem domain: while the scale of queries can vary, the concept remains the same.

Finally, attacker’s goal in this work was defined by three metrics: accuracy, fidelity, and transferability. We can view these metrics as (i) one measuring the performance of the substitute model on the task the target model was trained for, (ii) one measuring the similarity of performances of the target and substitute models, and (iii) one measuring the susceptibility of the target model to the adversarial examples crafted to fool the substitute model – or in other words, the similarity between the two models near their decision boundaries. However, the measurement of performance varies largely across tasks. For example, while in vision, accuracy is defined on a set of images, in trajectory prediction, it rather considers meters of deviation over a set of trajectories and their ground truth. This definition of success affects the attacker’s goal and how this goal is assessed, as in other attacks on AI (Grosse and Alahi, 2024). The framework may thus have to be adapted in terms of measurements of success as well. In general, however, the defined threat model can be adapted for other problem domains. Subsequently, our framework for attack comparison that follows from the threat model can also be transferred.

Best practices and open research questions. All of the recommendation (R1-R3) from best practices presented in Section 6 are generic and non-domain-specific and, therefore, should also be taken into account for attacks in other problem domains. The open research questions raised in Section 7 are derived from analysis of the particular group of attacks, and hence are more domain-specific. While some of these issues generalise across domains, and attacks in other problem domains are less established and systematised, the relevance of these questions for other domains should be studied separately.

9. Related Work

Below, we review influential works on evaluation methodology with similar contributions to other research fields.

Evaluation methodology is an important aspect in any discipline utilising experimentation, and as a field matures, research methodology becomes more precise with standards and best practices emerging. In security, the basis of any methodology is a solid threat model which is, ideally, practically motivated. Although previous surveys in model stealing (Oliynyk et al., 2023) have made an attempt to characterize model stealing attacks, they focus on the subtleties of different attacker’s goals like stealing functionality or model weights, and did not establish threat models. As we will see in this paper, the threat models discussed in literature that steal the model’s functionality vary greatly.

For other attacks on machine learning, such as evasion attacks, clear threat models have been established (Biggio and Roli, 2018). Based on these threat models, guidelines to evaluate defences (Carlini et al., 2019) based on adaptive attacks for adversarial examples (Carlini et al., 2019) or backdoors (Tan and Shokri, 2020) have been established. Such clear threat models allow to match attacks and defenses (Cinà et al., 2023) or generally compare or benchmark attacks within one threat model (Croce et al., 2021; Wu et al., 2022). Such a threat model is, for model stealing, to the best of our knowledge, currently missing; and there are no best practices regarding model stealing evaluations either.

While all fields within the area of ML security have received criticism for not evaluating on settings that are practical enough to be realistic (Apruzzese et al., 2023; Grosse et al., 2024), previous work has measured accessibility and granted queries required for model stealing attacks (Grosse et al., 2024) and shown that model stealing is perceived as a relevant threat by practitioners (Grosse et al., 2023). However, neither work surveys a precise threat model for model stealing, be it practical or not, as it occurs in academic works. We attempt to close this gap.

Orthogonal to our work, but listed for completeness, are approaches that identify wrong configurations, for example for evasion attacks (Pintor et al., 2022). Finding such failure cases for model stealing is beyond the scope of this work but would be, alongside benchmarks (Croce et al., 2021; Wu et al., 2022), highly useful for the field of model stealing.

10. Conclusion

Development of standardised evaluation methodology enables fair comparison among prior works, clear assessment of the current status of the research field, and, what is especially important in case of adversarial machine learning, facilitates countermeasure development. This work presents the first systematic effort for developing such methodology in the field of model stealing. We propose the first comprehensive threat model, and build the first attack comparison framework based on it. By extensively analysing prior attacks on image classification models, we derive best practical recommendations for designing, conducting, and evaluating model stealing attacks. Finally, we raise an in-depth set of research questions concerning evaluation methodology of model stealing attacks.

Acknowledgements.

The financial support by the Austrian Federal Ministry of Economy, Energy and Tourism, the National Foundation for Research, Technology and Development, the Christian Doppler Research Association and SBA Research (SBA-K1 NGC), a COMET Centre within the COMET – Competence Centers for Excellent Technologies Programme funded by BMIMI, BMWET, and the state of Vienna, managed by FFG, is gratefully acknowledged. This work has also received funding from the European Union’s Horizon Europe Research and Innovation Programme under grant agreement No 101136305. Partner Semmelweis University received funding from the Hungarian National Research, Development and Innovation Fund.

Appendix A Experiment Setup per Paper

Table 5 presents experiment setups from papers listed in Table 1.