A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

TL;DR

This paper introduces PartCrop, a unified and scalable membership inference method for visual self-supervised models that exploits part-aware responses, demonstrating its effectiveness across various training protocols and datasets, and proposing defenses and improvements.

Contribution

It proposes PartCrop, a novel unified membership inference approach that leverages part-aware responses, applicable across different self-supervised learning paradigms and scalable with structural enhancements.

Findings

PartCrop effectively infers membership across diverse models and training protocols.

Defenses like early stopping, differential privacy, and shrinking crop scale are effective.

Scaling experiments show the method's robustness and generalization in realistic scenarios.

Abstract

Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses within the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Finally, besides prototype testing on toy visual encoders and small-scale image datasets, we quantitatively study the impacts of scaling from both data and model aspects in a realistic scenario and propose a scalable PartCrop-v2 by introducing two structural improvements to PartCrop. Our code is at https://github.com/JiePKU/PartCrop.