TL;DR
This paper demonstrates the vulnerability of transfer-learned neural networks trained on small datasets to reconstruction attacks and introduces a novel, efficient homomorphic encryption scheme that protects training data without sacrificing model accuracy.
Contribution
It provides new attack methods for small-data transfer-learned classifiers and proposes a computationally efficient HE scheme that secures training data during inference.
Findings
DP-SGD fails to defend small-data transfer learning models against reconstruction attacks.
The proposed HE scheme effectively prevents data reconstruction without degrading classifier accuracy.
White-box and black-box attacks are thwarted by encrypting transfer-learned weights while keeping input data unencrypted.
Abstract
The growing body of literature on training-data reconstruction attacks raises significant concerns about deploying neural network classifiers trained on sensitive data. However, differentially private (DP) training (e.g. using DP-SGD) can defend against such attacks with large training datasets causing only minimal loss of network utility. Folklore, heuristics, and (albeit pessimistic) DP bounds suggest this fails for networks trained with small per-class datasets, yet to the best of our knowledge the literature offers no compelling evidence. We directly demonstrate this vulnerability by significantly extending reconstruction attack capabilities under a realistic adversary threat model for few-shot transfer learned image classifiers. We design new white-box and black-box attacks and find that DP-SGD is unable to defend against these without significant classifier utility loss. To…
Peer Reviews
Decision·Submitted to ICLR 2026
Attack contribution & evaluation. New hard-label black-box path (weight extraction → reconstructor) and a cleaner Neyman–Pearson ROC criterion for quantifying reconstruction at low FPR; results show sizable TPR at 1% FPR in few-shot TL. Pragmatic defense idea. RHE is conceptually simple (encrypt the TL head, not the inputs), and the prototype reports per-prediction latency that’s in the tens to hundreds of ms for small heads—reasonable for many TL scenarios.
Reading the title "Securing Transfer-Learned Networks with Reverse Homomorphic Encryption", I assume the paper is about a defense. However, most pages develop and measure the attack and the DP-SGD tradeoff. RHE is introduced later with a design sketch, implementation notes (CKKS/TenSEAL), and timing/accuracy tables, but there is no head-to-head evaluation demonstrating the attack failing under RHE (the defense claim is largely by construction: encrypt weights/logits ⇒ attacks can’t run). The def
1. They introduce new and effective reconstruction attacks (both white-box and a novel hard-label black-box) specifically tailored to a realistic adversary model in the few-shot transfer learning setting. 2. The authors show that the de facto defense, DP-SGD, fails to mitigate these attacks without incurring a severe, unacceptable loss in classifier utility. 3. The paper proposes a highly effective defense mechanism: Reverse Homomorphic Encryption (RHE). The properties of this defense are par
Please refer to my questions below.
1. This paper proposes effective white-box and black-box reconstruction attacks, which demonstrate stronger performance under realistic threat models compared to existing methods. 2. The experiments validate the applicability of the proposed attack methods on different datasets, showcasing their effectiveness and robustness against existing defense methods, such as DP-SGD. 3. The paper introduces the Reverse Homomorphic Encryption (RHE) mechanism, which effectively defends against data reconstru
1. The paper primarily focuses on the few-shot transfer learning scenario, but both the abstract and introduction fail to provide a systematic description of the research background. 2. The authors propose a novel reconstruction attack method, but there is a lack of detailed steps for the specific attack process. It appears to be a simple combination of several existing works, lacking novelty. 3. The paper proposes using Reverse Homomorphic Encryption (RHE) to defend against reconstruction attac
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsDepthwise Convolution · Pointwise Convolution · Batch Normalization · Depthwise Separable Convolution · Average Pooling · Inverted Residual Block · Sigmoid Activation · Global Average Pooling · 1x1 Convolution · Dropout
