Revisiting the Privacy Risks of Split Inference: A GAN-Based Data Reconstruction Attack via Progressive Feature Optimization

TL;DR

This paper introduces a GAN-based data reconstruction attack with progressive feature optimization that significantly improves privacy breach capabilities in split inference scenarios, especially on complex models and high-resolution data.

Contribution

It presents a novel GAN-based attack framework with hierarchical generator decomposition and semantic refinement, surpassing prior methods in effectiveness and generalizability.

Findings

Outperforms previous attacks in high-resolution scenarios

Effective against deep and complex DNNs

Works well in out-of-distribution settings

Abstract

The growing complexity of Deep Neural Networks (DNNs) has led to the adoption of Split Inference (SI), a collaborative paradigm that partitions computation between edge devices and the cloud to reduce latency and protect user privacy. However, recent advances in Data Reconstruction Attacks (DRAs) reveal that intermediate features exchanged in SI can be exploited to recover sensitive input data, posing significant privacy risks. Existing DRAs are typically effective only on shallow models and fail to fully leverage semantic priors, limiting their reconstruction quality and generalizability across datasets and model architectures. In this paper, we propose a novel GAN-based DRA framework with Progressive Feature Optimization (PFO), which decomposes the generator into hierarchical blocks and incrementally refines intermediate representations to enhance the semantic fidelity of reconstructed images. To stabilize the optimization and improve image realism, we introduce an L1-ball constraint during reconstruction. Extensive experiments show that our method outperforms prior attacks by a large margin, especially in high-resolution scenarios, out-of-distribution settings, and against deeper and more complex DNNs.

Tables

Table 1. TABLE I: Overall comparison of our method with state-of-the-art methods against ResNet-18 trained on CelebA.

Method	Split Point 1				Split Point 2				Split Point 3
	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
rMLE	24.084	0.023	0.731	0.266	21.294	0.044	0.603	0.413	14.101	0.242	0.310	0.642
LM	28.937	0.010	0.886	0.118	24.024	0.028	0.732	0.281	19.119	0.089	0.532	0.511
IN	25.128	0.021	0.764	0.241	22.950	0.034	0.694	0.279	20.755	0.059	0.642	0.292
GLASS	29.340	0.007	0.885	0.090	25.480	0.017	0.801	0.146	20.320	0.065	0.636	0.239
PFO(ours)	33.103	0.003	0.945	0.048	27.426	0.012	0.857	0.118	21.803	0.042	0.697	0.223

Table 2. TABLE II: Comparison results against ResNet-18 models trained on CelebA with the public dataset being FFHQ, i.e., the GAN prior is pre-trained on the FFHQ dataset.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	GLASS	27.89	0.010	0.848	0.123
	PFO(ours)	31.862	0.005	0.922	0.066
Split Point 2	GLASS	23.39	0.027	0.701	0.210
	PFO(ours)	24.883	0.020	0.769	0.186
Split Point 3	GLASS	17.85	0.108	0.453	0.336
	PFO(ours)	18.862	0.081	0.537	0.318

Table 3. TABLE III: Comparison results against the ResNet-152 model trained on CelebA with the public dataset being FFHQ. In this table, the resolution of images from CelebA and FFHQ is 224×224 and 256×256 respectively, serving as the high-resolution setting.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	GLASS	19.361	0.005	0.589	0.364
	PFO(ours)	21.647	0.003	0.711	0.240
Split Point 2	GLASS	17.039	0.010	0.550	0.412
	PFO(ours)	19.496	0.005	0.658	0.311
Split Point 3	GLASS	15.823	0.013	0.494	0.459
	PFO(ours)	16.023	0.012	0.560	0.415

Table 4. TABLE IV: Comparison results against the ResNet-152 model trained on FaceScrub with the public dataset being FFHQ. As the high-resolution setting, the resolution of images from FaceScrub and FFHQ is 224×224 and 256×256, respectively.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	GLASS	20.003	0.005	0.665	0.406
	PFO(ours)	23.012	0.003	0.795	0.265
Split Point 2	GLASS	16.087	0.013	0.610	0.485
	PFO(ours)	19.423	0.006	0.747	0.337
Split Point 3	GLASS	15.069	0.015	0.545	0.516
	PFO(ours)	15.568	0.013	0.631	0.434

Table 5. TABLE V: Comparison results against the CLIP-RN50 model. Both of the target model and the GAN prior are trained on the FFHQ dataset (private and public parts), with the resolution set to 256×256.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	GLASS	15.227	0.015	0.650	0.368
	PFO(ours)	16.014	0.013	0.825	0.150
Split Point 2	GLASS	14.010	0.020	0.600	0.408
	PFO(ours)	15.136	0.016	0.786	0.170
Split Point 3	GLASS	14.466	0.018	0.584	0.384
	PFO(ours)	16.582	0.012	0.743	0.168

Table 6. TABLE VI: Comparison results against the ResNet-18 model trained on the CelebA dataset under the black-box setting.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	GLASS	18.366	0.090	0.564	0.268
	PFO(ours)	19.263	0.072	0.591	0.261
Split Point 2	GLASS	16.667	0.146	0.491	0.312
	PFO(ours)	17.502	0.115	0.534	0.287
Split Point 3	GLASS	13.378	0.302	0.346	0.371
	PFO(ours)	14.316	0.260	0.393	0.357

Table 7. TABLE VII: Comparison results against the ResNet-18 model trained on CINIC-10.The StyleGAN-XL prior is trained on CIFAR-10.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	GLASS	15.249	0.706	0.258	0.369
	PFO(ours)	16.801	0.488	0.362	0.292
Split Point 2	GLASS	14.436	0.857	0.223	0.375
	PFO(ours)	14.937	0.767	0.256	0.327
Split Point 3	GLASS	13.683	1.083	0.154	0.435
	PFO(ours)	13.972	0.976	0.162	0.412

Table 8. TABLE VIII: Comparison results against the ResNet-18 model trained on CelebA with defenses. The experiments in this table are performed on Split Point 1.

Defense Strategy	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
No Peek	GLASS	21.160	0.091	0.701	0.233
	PFO(ours)	22.441	0.052	0.785	0.195
DISCO	GLASS	28.905	0.008	0.879	0.099
	PFO(ours)	31.236	0.005	0.922	0.068
Siamese	GLASS	28.847	0.011	0.889	0.108
	PFO(ours)	29.060	0.009	0.929	0.079

Table 9. TABLE IX: Ablation study performed on ResNet-18 trained on the CelebA dataset. ”w/o” is the abbreviation for ”without”.

Split Point	Method	PSNR$\uparrow$	MSE$\downarrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Split Point 1	w/o PFO	28.292	0.009	0.867	0.106
	PFO w/o $l_1$-ball	32.184	0.005	0.918	0.075
	PFO	33.103	0.003	0.945	0.048
Split Point 2	w/o PFO	24.523	0.022	0.767	0.168
	PFO w/o $l_1$-ball	24.449	0.022	0.786	0.207
	PFO	27.426	0.012	0.857	0.118
Split Point 3	w/o PFO	18.180	0.113	0.549	0.301
	PFO w/o $l_1$-ball	18.714	0.083	0.587	0.348
	PFO	21.803	0.042	0.697	0.223

Equations

$${\text{x}}^{\ast }=\underset{\text{x}\in \mathcal{X}}{\mathrm{argmin}}{d}_{\mathcal{H}}(M_C(\text{x}),{\text{h}}_{tar})+\lambda R(\text{x}),$$

$$M_{best}^{-1}=\underset{M^{-1}}{\mathrm{argmin}}{d}_{\mathcal{X}}(M^{-1}(M_C({\text{x}}_{pub})),{\text{x}}_{pub})),$$

$${\text{z}}^{\ast }=\underset{\mathbf{z}\in \mathcal{Z}}{\mathrm{argmin}}{L}_{match}(M_C({\text{x}}_{tar}),M_C(G(\text{z})))$$

$$+\lambda R(\text{z})+\alpha \mathcal{TV}(G(\text{z})),$$

$$R(\text{z})=-\frac{1}{2}\sum _{i=1}^k\left(1+\log ({\sigma }_i^2)-{\mu }_i^2-{\sigma }_i^2\right),$$

$$\mathcal{TV}(\text{x})=\sum _{i,j}\sqrt{|{\text{x}}_{i+1,j}-{\text{x}}_{i,j}{|}^2+|{\text{x}}_{i,j+1}-{\text{x}}_{i,j}{|}^2},$$

$$\mathcal{L}=L_{match}({\text{h}}_{tar},{\text{h}}_{rec})+\alpha \mathcal{TV}(G_{remain}(\text{hf},\text{w})),$$

Full text

I Introduction

In recent years, Deep Neural Networks (DNNs) have undergone rapid development and achieved tremendous success in various applications, such as face recognition [1], personalized recommendation [2], and privacy computing [3]. However, alongside the continuous enhancement in model performance, the growing size of model parameters has become a burden for edge devices. As a result, it is challenging to deploy these models on resource-constrained edge devices while satisfying efficiency requirements (e.g., computational cost and response delay). Meanwhile, users utilizing cloud APIs raise significant concerns about privacy leakage, as raw data is collected by cloud service providers for service provisioning.

In this context, Split Inference (SI) [4, 5, 6, 7, 8, 9] has emerged as a promising solution since it balances efficiency with privacy. Specifically, SI enables model providers to partition a large-scale DNN into two segments, deploying them on edge devices and cloud servers, respectively, to facilitate collaborative inference between the cloud and edge devices [9]. Each user’s private raw data undergoes preprocessing on the partial model at the edge device before the corresponding intermediate representations are transmitted to the cloud server for subsequent inference. Therefore, SI prevents cloud service providers from accessing raw data directly, ensuring the privacy and security of users. Moreover, as both the edge devices and cloud server execute only a portion of the full model, SI alleviates resource constraints on edge devices while fully leveraging the abundant storage and computational resources of the cloud server.

Despite the aforementioned advantages, recent studies [10, 11, 12, 13, 14, 15, 16, 17, 18, 19] have revealed that SI systems still pose threats of privacy leakage. Among these attack methods, the Data Reconstruction Attack (DRA) [20, 21, 22, 23, 24] is known as a representative threat due to its powerful capability in recovering the privacy-sensitive data. As shown in Fig. 1, malicious attackers (e.g., a curious service provider who desires to obtain the privacy from users) aim to exploit the intermediate representations exchanged between the cloud and edge devices to reconstruct the user’s input data, thereby recovering the targeted privacy-sensitive information. Given that DRAs can be launched throughout the entire model release lifecycle and achieve pixel-level image reconstruction, they tend to pose a more threatening nature to users’ privacy.

However, existing DRA methods suffer from several limitations that significantly diminish their effectiveness and practicality. For instance, current DRAs [20, 21, 22, 23, 24] can only attack shallow models with simple architectures (e.g., LeNet [25]) and reconstruct images with relatively low resolution (i.e., $64\times 64$ pixels), falling short of adapting to more complex model architectures and higher-resolution images encountered in real-world scenarios. Moreover, these DRAs exhibit weak support for Out-of-Distribution (OOD) scenarios, where there is a significant distributional shift between the target private dataset and the public dataset utilized as auxiliary prior [20]. The primary reason for the current limitation is that most DRAs fail to sufficiently leverage the prior knowledge available from public datasets as auxiliary information. Although certain DRA methods like [23] introduce a Generative Adversarial Network (GAN) pre-trained on the public dataset as an image prior, they merely explore its latent space coarsely and fail to exploit its full potential.

To address this, we propose a novel GAN-based DRA approach, Data Reconstruction Attack via Progressive Feature Optimization (PFO), which effectively disassembles the GAN structure and leverages hierarchical features between intermediate blocks. This intuition originates from recent studies [26, 27, 28, 29] that have demonstrated that there is rich semantic information encoded in the latent code and hierarchical features of GANs. Specifically, we consider the generator of the GAN as a concatenation of multiple basic blocks and the representations produced between the blocks as hierarchical features. To launch the attack, we first perform optimization in the latent space of the generator and then successively optimize the hierarchical features from the first internal block to the last internal block. Furthermore, we introduce an $l_{1}$ ball constraint to restrict the deviation during optimization, to avoid unreal image generation. During this process, the reconstructed image will progressively approximate the macro-features and the fine-grained features of target images, ultimately achieving pixel-level reconstruction of the privacy-sensitive images. We conduct systematic experiments to evaluate our method at different split points in multiple settings, including high-resolution image recovery, OOD scenarios, and various target models. Moreover, we also conduct assessments against defense strategies and extend our attack to the black-box scenario and heterogeneous data. The encouraging experimental results demonstrate that our proposed method outperforms previous DRA methods on multiple metrics and maintains robustness against the employed defense mechanisms compared to baselines. Our main contributions are summarized as follows:

• •

We propose a novel GAN-based DRA method named PFO, which disassembles the generator into basic blocks and progressively optimizes the latent code and hierarchical features under the $l_{1}$ ball constraint.

• •

The proposed attack achieves state-of-the-art performance in a range of scenarios, particularly under the high-resolution image recovery, out-of-distribution (OOD) settings, and models with more complex structures.

• •

We conduct extensive experiments to validate the robustness of our method against defense strategies and outstanding transferability on heterogeneous data and black-box scenarios.

II Background

II-A Split Inference

Split Inference (SI) evolved from Split Learning (SL) [4, 5, 6, 7, 8, 9], aiming to distributively deploy the inference process of structurally complex models across cloud servers and edge devices. This technique simultaneously reduces computational overhead on the endpoint devices and ensures data privacy to some extent, thus exhibiting considerable application potential.

In a typical SI system, a well-trained model $M$ is partitioned into two parts, which are deployed as client model $M_{C}:\mathcal{X}\xrightarrow{}\mathcal{H}$ for users and cloud model $M_{S}:\mathcal{H}\xrightarrow{}\mathcal{Y}$ for service providers, respectively. When users input their raw data x into the SI system, the client model $M_{C}$ first processes it to obtain the intermediate representation $\textbf{h}=M_{C}(\textbf{x})$. Then, h is transmitted to the cloud server and mapped into the proper output $y$ through the cloud model $y=M_{S}(\textbf{h})$. Finally, the output $y$ is returned to the users to complete the whole inference process. Through collaborative inference between the cloud and the client, the edge-side devices merely execute the client model $M_{C}$ instead of the full model $M$, facilitating the reduction of computation overhead. Meanwhile, the cloud server only has access to the smashed intermediate representation h, providing a certain level of privacy preservation for users. By incorporating the above two advantages, the SI technique is capable of striking a trade-off between privacy and utility.

II-B Data Reconstruction Attacks on Split Inference

Data Reconstruction Attacks (DRAs) enable pixel-level reconstruction of privacy-sensitive features in target images, as shown in Fig.1. Typically, DRAs can be categorized as Optimization-based methods and Learning-based methods according to attacking mechanisms.

For the Optimization-based DRAs, the adversary tries to optimize an initial image x sampled from a random distribution or a pre-trained generator. Since the attacker can only access intermediate representations, the optimization objective for this problem is defined as minimizing the distance between the representations computed from the initial image $\textbf{h}=M_{C}(\textbf{x})$ and the target representations $\textbf{h}_{tar}$. The optimization problem is formulated as follows:

[TABLE]

where $d_{\mathcal{H}}(.,.)$ represents the specific metric for measuring the distance between representations, $R(.)$ serves as an auxiliary regularization term to ensure realistic generation of the initial image, and $\lambda$ is a hyperparameter controlling the weight of $R(.)$. Through this optimization process, the final image $\textbf{x}^{*}$ maximally approximates the target image $\textbf{x}_{tar}$, i.e., $\textbf{x}^{*}\approx\textbf{x}_{tar}$.

The pioneering work, rMLE [20], first formulated this attack as an optimization problem, minimizing the Euclidean distance between feature representations while using Total Variation (TV) [30] as a prior to ensure the image’s visual plausibility. Building on this, Likelihood Maximization (LM) [21] significantly improved reconstruction quality by employing a pre-trained autoencoder as a more powerful deep image prior, which constrains the generated image to a more natural manifold.

For the Learning-based DRAs, the adversary aims at training an inverse network $M^{-1}:\mathcal{H}\xrightarrow{}\mathcal{X}$ to directly invert $\textbf{h}_{tar}$ back to the image space $\mathcal{X}:\textbf{x}^{*}=M^{-1}(\textbf{h}_{tar})$. To obtain $M^{-1}$, the attacker tends to utilize a publicly available dataset $D_{pub}$ and process it through the $M_{C}$ to get a paired dataset $(\textbf{x}_{pub},M_{C}(\textbf{x}_{pub}))$. Then, $M^{-1}$ is trained as follows:

[TABLE]

where $d_{\mathcal{X}}(.,.)$ measures the distance between the images. After training, the attacker obtains the final image using the $M^{-1}_{best}$, i.e., $\textbf{x}^{*}=M^{-1}_{best}(\textbf{h}_{tar})$.

A representative method is the Inverse-Network (IN) [20]. This approach involves training a dedicated decoder on feature-image pairs acquired from the client model, which learns a direct mapping that can reconstruct an image from a target representation in a single forward pass.

To counter such threats, various defenses have been developed to protect intermediate representations. Early strategies focused on feature obfuscation; for instance, Siamese Defense [31] employs a Siamese network to learn inversion-resistant features, while NoPeek [32] utilizes adversarial training to make representations uninformative to an attacker. Subsequent methods shifted towards direct information reduction, such as DISCO [21] which sparsifies feature maps.

II-C GAN as prior

Generative Adversarial Networks (GANs) [33] are a class of DNNs composed of a generator and a discriminator. Through adversarial training, they learn the distribution from the training dataset with the objective of generating high-quality and diverse realistic images [34]. Currently, prominent examples of GANs include the WGAN [35] and StyleGAN series [36, 37, 38]. With strong generative capabilities, GANs have been widely used in various privacy attacks [39, 40, 41, 42, 43, 44], and were first introduced into the DRA field by GLASS [23].

Recent works [26, 27, 28, 29] have demonstrated the rich semantic information of hierarchical features in GANs, indicating the potential to go deep into the internal structure. Thus, our investigation focuses on exploring the possibility of leveraging the hierarchical space of internal blocks to enhance DRA.

III Methodology

In this section, we begin by explaining the threat model of our PFO method in the white-box SI system and introducing the GAN prior employed in our method. Subsequently, we elaborate on the detailed pipeline of the proposed PFO step by step. The overview of our PFO method is shown in Fig. 2. Finally, we further discuss how to migrate the PFO method to the black-box scenario.

III-A Preliminaries

Threat model. Following previous works [20, 21, 22, 23, 24], we assume the cloud service provider acts as an honest-but-curious member in the SI system, whose goal is to reconstruct the target private image $\textbf{x}_{tar}$ from the received intermediate representations $\textbf{h}_{tar}$. Considering that service providers tend to possess greater privileges in real-world scenarios, we account for the worst-case setting where the attacker can gain white-box access to the full parameters and architecture of the target model $M$ (including the client part $M_{C}$ and the server part $M_{S}$). Besides, the Internet hosts a vast number of open-source datasets, where the adversary can readily retrieve essential datasets $D_{pub}$ with similar distribution to the target dataset $D_{tar}$ (e.g., selecting other facial datasets as auxiliary datasets when attacking a facial recognition dataset) and use them to train generative priors. In this paper, our proposed PFO is an optimization-based DRA method and utilizes a GAN prior to provide high-fidelity images. Therefore, the overall optimization process of PFO follows Eq. (1) and the optimized image x is produced by the GAN prior $\textbf{x}=G(\textbf{z})$, where z is the input latent vector of the generator.

GAN prior. Our PFO method relies on the pre-trained StyleGAN2 [37] as a generative prior. The generator $G$ of StyleGAN2 consists of two components, which are a mapping network $G_{map}:\mathcal{Z}\xrightarrow{}\mathcal{W}$ and a synthesis network $G_{syn}:\mathcal{W}\xrightarrow{}\mathcal{X}$, respectively. The mapping network $G_{map}$ maps the initial latent vectors $\mathbf{z}\in{\mathcal{Z}}$ into the extended $\mathcal{W}$ space [45], while the synthesis network $G_{syn}$ generates images $\mathbf{x}$ using the extended vectors $\mathbf{w}\in{\mathcal{W}}$. According to the previous research [45], the reduced feature entanglement in $\mathcal{W}$ space facilitates better style generation. Thus, we also optimize $\mathbf{w}$ during the progressive feature optimization. Notably, the StyleGAN2 is pre-trained without the utilization of the target model $M$ or other auxiliary prior corresponding to the target dataset, making it more flexible when attacking different target models and datasets under OOD scenarios.

III-B Data Reconstruction Attack via Progressive Feature Optimization

Initial Selection. Considering the randomness in sampling latent vectors $\mathbf{z}$, it is obvious that not all of them facilitate the generation of appropriate images. However, for the optimization of non-convex functions, a proper initial point is extremely significant to determine the whole optimization direction. To mitigate the risk of generating low-quality and misleading images, it is critical to perform initial selection in $\mathcal{Z}$ space and $\mathcal{W}$ space to avoid local minima and obtain reliable latent vectors. Specifically, we first randomly sample a large number of $\mathbf{z}$ as candidates and then optimize them as follows:

[TABLE]

where $L_{match}(.,.)$ is the Mean Square Error (MSE) loss between the target representation and the representation computed from the generated image $G(\textbf{z})$, serving as the main guidance for z. $\lambda$ and $\alpha$ are hyperparameters to determine the weight of regularization terms, respectively. To avoid unreal generation when using MSE loss, we introduce KL-based regularization [46] to constrain the z to ensure it conforms to the standard Gaussian distribution:

[TABLE]

where $\mu_{i}^{2}$ and $\sigma_{i}^{2}$ are the element-wise mean and standard deviation of the z. Additionally, we adopt Total Variation [30] to encourage piece-wise smooth generation as follows:

[TABLE]

where $\textbf{x}=G(\textbf{z})$. By combining the above loss terms, it can be ensured that the final $\textbf{z}^{*}$ is a relatively ideal starting point to generate the initial extended vector $\textbf{w}_{init}=G_{map}(\textbf{z}^{*})$ for the following stage.

Progressive Feature Optimization. Previous research [37] proves that the front blocks in the StyleGAN2 control the overall characteristics, while the back ones concentrate on local details. Therefore, we can leverage the internal blocks of the generator to achieve progressive optimization. The complete workflow of progressive feature optimization is displayed in Algorithm 1.

First, we optimize $\textbf{w}_{init}$ to obtain the optimal start point $\textbf{w}_{0}$. The synthesis network $G_{syn}$ is split into $H+1$ internal blocks, i.e., $G_{syn}=G_{H+1}\circ{G}_{H}\circ\cdots{G}_{2}\circ{G}_{1}$. Each block contains two styled convolution modules [36], which are the basic generation units in the StyleGAN2. Then, we may input the $\textbf{w}_{0}$ into the first block $G_{1}$ to obtain the first hierarchical feature $\mathbf{hf}_{1}^{0}$. For each internal block $G_{i+1},i\in[1,\dots,H]$, its hierarchical feature $\mathbf{hf}_{i+1}^{0}$ is computed by following steps:

• •

Initially, we generate images utilizing the remaining blocks ($i.e.,\mathbf{x}_{i}^{j-1}={G}_{remain}(\mathbf{hf}_{i}^{j-1},\mathbf{w}_{i}^{j-1})$) and feed them into $M_{C}$ to compute the loss with the target intermediate representation $\textbf{h}_{tar}$.

• •

Then, we repeat the former step to iteratively update both $\mathbf{w}_{i}$ and $\mathbf{hf}_{i}$. Meanwhile, we restrict the $\mathbf{hf}_{i}$ within the $l_{1}$ ball with radius ${r}[{i}]$ centered at the initial hierarchical feature $\mathbf{hf}_{i}^{0}$, which acts as a correction device to avoid exceeding shift that may lead to collapse generation.

• •

Once the iteration is finished, the final $\mathbf{w}^{N}_{i}$ and $\mathbf{hf}^{N}_{i}$ will be fed into the next block $G_{i+1}$ for the next hierarchical feature $\mathbf{hf}_{i+1}^{0}$. Notaly, we denote the $\mathbf{w}^{N}_{i}$ as the initial vector $\mathbf{w}_{i+1}^{0}$ before the next optimization starts.

After all the hierarchical features are optimized, the final image $\textbf{x}^{*}$ can be obtained by $\textbf{x}^{*}=\textbf{hf}_{H+1}^{0}=G_{H+1}(\textbf{hf}_{H}^{N})$.

Since an $l_{1}$-ball constraint is employed during the progressive feature optimization to constrain the distribution of extended vectors w and hierarchical feature hf, we remove the KL divergence term from the loss function in this stage. Hence, the loss function $\mathcal{L}$ is formulated as follows:

[TABLE]

where $\textbf{h}_{rec}=M_{C}(G_{remain}(\textbf{hf},\textbf{w}))$.

III-C Extension to Blackbox Scenario

In the black-box setting, the malicious attacker loses access to the model’s parameters and architectures, retaining only the ability to query it for intermediate representations. At this point, the adversary is unable to compute gradients to perform backpropagation, thus leading to the urgent need for seeking gradient-free optimization methods.

For our proposed PFO method, we select the typical Covariance Matrix Adaptation (CMA) [47] optimizer as an alternative to gradient-based optimizers. This optimizer employs gradient-free search to estimate the direction for parameter optimization, thus allowing our approach to execute under black-box settings. The corresponding experiment in Sec. IV-D shows that our method maintains the best performance compared to baselines under the black-box scenario.

IV Experiments

In this section, we conduct comprehensive experiments to validate the effectiveness of our proposed method. We first illustrate the details of our experimental settings. Then, we make an overall comparison between our method and state-of-the-art baselines to evaluate the attack performance. Furthermore, we conduct extensive experiments on more challenging scenarios to assess the generalizability of our method in various settings. Finally, we perform an ablation study to assess the contributions of each component in our method.

IV-A Experimental Setup

Models. We evaluate our attack against a diverse set of target models ($M_{C}$), including the ResNet-18, ResNet-152 [1] and the CLIP-RN50 vision-language model [48]. These models are assumed to be privately trained and deployed by a service provider. For the image reconstruction, we employ a pre-trained StyleGAN2 [37] as a strong generative prior, and we utilize StyleGAN-XL [49] in heterogeneous scenarios.

Datasets. We utilize (1) CelebA [50], which contains 202,599 face images of 10,177 identities (2) FFHQ [36], which consists of 70,000 high-quality face images with considerable variation in terms of age, ethnicity, and image background. We conduct facial reconstruction experiments at two scales: for standard-resolution ($64\times 64$) analysis, the downstream task is binary attractiveness classification on CelebA; for high-resolution ($224\times 224$) analysis, the task is 1000-class classification. Furthermore, to evaluate performance on heterogeneous data, we employ (3) CIFAR-10 [51] and (4) CINIC-10 [52], which are established 10-class object recognition datasets.

Split Point. As shown in Fig. 3, we divide the target model into six blocks and present three split points. The blocks before the split point are treated as the client model $M_{C}$ and the blocks after the split point serve as the cloud model $M_{S}$, respectively.

Compared Baselines & Defenses. We benchmark our method against four prominent data reconstruction attacks (DRAs): regularized Maximum Likelihood Estimation (rMLE) [20], Likelihood Maximization (LM) [21], Inverse-Network (IN) [20], GLASS [23]. To further assess the robustness of our approach, we evaluate its performance against four representative privacy-preserving defenses, including Siamese Defense [31], NoPeek [32], DISCO [21], and Noise Mask [53].

Attack Setup. Our experimental setup involves distinct private and public datasets. For experiments on CelebA, we partition it into a private set $D_{P}$ (80,525 images from 3,000 identities) for training the target model, and a public set $D_{A}$ (the remaining images) for training the StyleGAN2. This ensures no identity overlap between the datasets. To evaluate transferability, we also conduct a cross-dataset attack where the StyleGAN2 is trained on the public FFHQ dataset and used to attack the model trained on CelebA.

We configure the hyperparameters for our attacks and the baselines to ensure a fair comparison. For optimization-based methods, including our own, the learning rate is set to $1\times 10^{-2}$ for 20,000 iterations. The TV regularization for rMLE and LM is set to 2.0 and 1.5, respectively, while for our GAN-based attack it is set to 0.01. For the learning-based IN, the inversion model is trained for 30 epochs with a learning rate of $1\times 10^{-3}$.

When evaluating against defenses, we select a challenging hyperparameter setting for each mechanism. For DISCO, we use a privacy-utility trade-off $\rho=0.95$ and a channel pruning ratio of $R=0.1$. For Noise Mask, we inject Laplacian noise with location $a=0$ and scale $b=1.0$. The trade-off parameter for NoPeek is set to $\lambda_{2}=5$, and for Siamese Defense, the weight of the Siamese loss is set to $\lambda_{3}=0.005$.

Evaluation Metrics We evaluate the similarity between the original and reconstructed images using four complementary metrics to comprehensively quantify the reconstruction performance. We employ Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR) [54] to measure pixel-wise fidelity. To better align with human perceptual judgment, which is crucial for assessing privacy risks, we also adopt the Structural Similarity Index (SSIM) [55] and the Learned Perceptual Image Patch Similarity (LPIPS) [56]. For MSE and LPIPS, lower values indicate a more successful reconstruction, whereas for PSNR and SSIM, higher values are better.

IV-B Overall Comparison on Attack Performance

We compare our PFO method with the previous state-of-the-art methods, which are rMLE [20], LM [21], IN [20], and GLASS [23]. All the hyperparameters of the compared baselines are aligned with the recommended values in their official papers.

The overall comparison results are listed in the Table I. The target model is ResNet-18 trained on the private set of CelebA and the generative prior is pre-trained on the public set of CelebA. The resolution of images from the public dataset and private dataset is set to 64×64, following the common settings in the previous works [20, 21, 22, 23]. From the quantitative results in Table I, we can observe that our PFO method outperforms all the baselines on all four metrics under the three split points. Taking the PSNR metric as an example, our method achieves an improvement of at least 1.5 dB and up to 4 dB over the previous state-of-the-art methods, highlighting the superiority of the PFO method.

Visualization of the reconstructed images is displayed in Fig. 4. Our reconstructed images demonstrate higher fidelity and realism thwan baselines, further validating the superiority of exploiting GAN’s intermediate features.

IV-C Evaluation in Challenging Scenarios

In this part, we further investigate the effectiveness of our method in more challenging scenarios. Due to the page limit, subsequent tables will only present experimental results for our PFO method and GLASS. The reason why we select GLASS as the baseline is that it is the state-of-the-art method among prior approaches as shown in the Table I.

Out-of-Distribution (OOD) Settings. Unlike the common setting [23] where the public dataset shares a similar distribution with the private dataset (e.g., dividing the CelebA dataset into disjoint parts), the OOD setting indicates that the two datasets have large distribution shift with each other. Specifically, the two datasets are totally different datasets instead of two slices from a single large dataset. The corresponding results are shown in Table II, III and IV.

Table II shows the evaluation results of low-resolution (64×64) OOD settings. Compared to Table I, our PFO method and GLASS both suffer a decrease in the overall performance, while PFO method still maintains the best performance. Table III and IV display the evaluation results of high-resolution (224×224) OOD settings. It is obvious that our PFO presents better transferability than the baseline in both high-resolution and low-resolution scenarios.

High-resolution Image Recovery Settings. Previous works [20, 21, 22, 23] merely concentrate on the easier low-resolution (64×64) scenarios. Therefore, we increase the resolution to 224×224 to better suit real-world applications. The experimental results for high-resolution evaluation are listed in Table III, IV, and V.

Table III and IV validate the high-resolution image recovery under the harder OOD setting, while Table V assesses the performance under the non-OOD setting. The results presented in the three tables demonstrate that our method achieves the best under the high-resolution settings.,

Assessment of More Complex DNNs. Except for the commonly used ResNet-18 model in previous works [20, 21, 22, 23], we further employ complex models for evaluation, including ResNet-152 and CLIP-RN50. Related experimental results are shown in Table III, IV, and V.

According to the results in the aforementioned three tables, our PFO method is excellent in adapting to the complex models. However, we notice that the CLIP-RN50 model causes the most reduction in the performance even in the non-OOD scenario. The potential reason for this phenomenon might be the different tasks where the model is trained, e.g., the CLIP-RN50 is trained as a vision foundation for multiple vision tasks, while the ResNet-152 is trained for simple classification tasks. This highlight indicates the promising future work on how the tasks affect the performance of DRAs.

IV-D Extended Evaluation

Apart from the above evaluation, we conduct more experiments to comprehensively validate the generalizability of our PFO method. By default, the target model is the ResNet-18 trained on the private set of CelebA and the GAn prior is pre-trained on the public set of CelebA, which are the same as the main experiment of Table I.

Defense Analysis. Following prior works [23], we select three typical defenses against DRAs to analyze the robustness of our proposed PFO, which are NoPeek [32], Siamese Defense [31] and DISCO [21]. The results are listed in the Table VIII. Due to the limited paper length, we only present the evaluation results in the Split Point 1.

As shown in Table VIII, our PFO method maintains the superior robustness against the three defenses, compared to the baseline. Notably, the DISCO and Siamese defenses are weak in resisting the current state-of-the-art DRAs, while the NoPeek still maintains effectiveness to some extent. This result indicates that with the continuous improvement in DRA performance, stronger defenses accordingly need to be continuously studied and updated.

Black-box Setting. Under the black-box scenario, the adversary has no access to the parameters and detailed architectures of the target model, thereby unable to compute the gradients for backpropagation. We substitute the gradient-based optimizer with gradient-free optimizer CMA [47] to further evaluate whether the proposed method can be effective under the black-box scenario.

Table IV-D demonstrates the evaluation results of black-box attacks. Although both methods suffer a significant decrease in the attack performance, our method still outperforms the baseline method GLASS. This result reveals the challenges of black-box scenarios, shedding light on future research directions.

Heterogeneous Data. The above experiments are all conducted on the facial datasets. Following the prior work [23], we extend our evaluation to the heterogeneous data, such as natural images in CINIC-10 dataset. The results are listed in the Table VII. Our PFO demonstrates better performance than the baseline method.

Visualization of heterogeneous data is shown in Fig .5.

IV-E Ablation Study

To estimate the contributions from each component in our method, we conduct ablation studies on the ResNet-18 trained on the private set of CelebA dataset using the StyleGAN2 trained on the public dataset of CelebA. The overall results are listed in the Table IX.

Progressive Feature Optimization. We remove the progressive feature optimization stage from our pipeline while keeping the remaining paradigm unchanged. The corresponding setting is named ”w/o PFO”. From the results between ”w/o PFO” and PFO, we can observe that there is a significant degradation up to 4 dB in PSNR metric. Other metrics have also shown varying degrees of decline. Therefore, utilizing the hierarchical features plays a critical role in our pipeline.

$l_{1}$-ball constraint. To avoid unreal image generation, we introduce the $l_{1}$-ball constraint into the progressive features optimization. The setting of removing $l_{1}$-ball is named ”PFO w/o $l_{1}$-ball”. As shown in Table IX, the $l_{1}$-ball is beneficial in improving the performance in all metrics. Thus, we demonstrate the necessity of restricting the hierarchical features and w vectors within the $l_{1}$-ball space.

V Conclusion and Future Work

In this paper, we introduce PFO, a novel and strong data reconstruction attack designed for split inference systems. Through extensive experiments, we demonstrate that PFO consistently outperforms existing methods. Its efficacy has been validated across a wide spectrum of challenging scenarios, including against prominent defenses like NoPeek and DISCO, in challenging high-resolution settings, and even in cross-model and black-box configurations. These findings reveal profound privacy vulnerabilities in modern deep learning models deployed in split inference frameworks and highlight that current defense mechanisms may offer insufficient protection against powerful generative attacks.

While PFO sets a new benchmark for reconstruction fidelity, we identify its computational overhead—a common trait of optimization-based methods—as a key limitation. This presents clear avenues for future research. An important direction may involve enhancing the attack’s efficiency, perhaps by developing hybrid models that integrate our optimization strategy with faster learning-based techniques. Furthermore, our findings highlight the urgent need for novel defense mechanisms specifically designed to counteract attacks that exploit powerful generative priors. Additionally, extending the evaluation of PFO to other emerging model architectures, such as Vision Transformers, will be crucial for comprehensively understanding and mitigating the evolving privacy risks in next-generation split inference systems.

Societal Impact and Ethical Considerations. A potential negative impact could be malicious users leveraging the proposed attack method to reconstruct private data from the public system. To alleviate potential dilemma, a cautious approach is to establish access permissions.