Robust Representation Consistency Model via Contrastive Denoising
Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar
TL;DR
This paper introduces a novel contrastive denoising approach in latent space that enhances robustness of neural networks against adversarial attacks, achieving state-of-the-art accuracy with significantly reduced inference costs.
Contribution
It reformulates diffusion-based robustness certification as a discriminative task in latent space, enabling implicit denoising and classification with minimal computational overhead.
Findings
Outperforms diffusion-based methods on ImageNet across all perturbation radii.
Achieves up to 11.6% higher certified accuracy at larger radii.
Reduces inference costs by 85 times compared to existing diffusion methods.
Abstract
Robustness is essential for deep neural networks, especially in security-sensitive applications. To this end, randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations. Recently, diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples before making predictions with a standard classifier. While these methods excel at small perturbation radii, they struggle with larger perturbations and incur a significant computational overhead during inference compared to classical methods. To address this, we reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space. Specifically, we use instance discrimination to achieve consistent representations along the trajectories by aligning temporally adjacent points. After…
Peer Reviews
Decision·ICLR 2025 Poster
- The paper introduces a structured noise schedule for diffusion-based Randomized Smoothing, reformulating de-noising from a generative to a discriminative task. This is an interesting idea, especially in adversarial robustness, where noise schedules and latent consistency are less commonly used. - The authors perform a thorough evaluation of their approach, comparing it against multiple state-of-the-art methodologies, in terms of certified accuracy and inference time.
- Motivation: the main idea of Carlini et al. (2022) is to utilize an existing diffusion model for de-noising, thus getting randomized smoothing "for free", without needing to fine-tune the base classifier. This work reformulates the diffusion process to be more aligned in the latent space, but in the end, we get a classifier that's trained from scratch, with a diffusion-like objective. Why do all that and not just train a classifier on noisy images (since we're going to train in any case)? What
+ The overall structure and writing approach of the paper are well-organized and clear. + Combining consistency model with robust representation learning is interesting and novel to me. The use of PF-ODE to build a consistent relationship between clean data and perturbed data has a theoretical basis. + The proposed method is significantly more time-efficient than previous diffusion-based methods, both theoretically and practically..
- Some details of the method are unclear; it would be helpful to provide the complete loss function formula used in the approach, along with a detailed explanation. - The improvements in Certified Accuracy shown in Table 1 are marginal, suggesting limited gains in robustness compared to prior methods.
1. The proposed method makes sense intuitively and seems like a novel way to bake adversarial robustness into representation learning pipelines 2. Experimental evaluations are fairly thorough and the performance improvements over prior methods are quite large.
1. I can’t quite understand what is the message behind the image generation experiment in section 4.5. Is the diffusion model trained on actual images in CIFAR10? If so, then regardless of whether the conditioning encoder extracts meaningful semantic representations, the diffusion model will produce images that resemble CIFAR10 images. 2. Overall, writing could use some polish. Some examples: * Line 42 “While empirical defenses train DNNs to be robust to known adversarial examples (Madry et a
Code & Models
Videos
Taxonomy
TopicsComplex Network Analysis Techniques
MethodsDiffusion · Randomized Smoothing