Randomized Smoothing under Attack: How Good is it in Pratice?

TL;DR

This paper critically evaluates the practical robustness of randomized smoothing against black-box attacks, revealing a significant gap between theoretical guarantees and real-world effectiveness.

Contribution

It highlights the mismatch between theoretical certification and practical attack scenarios, providing empirical evidence of randomized smoothing's limitations in real-world defenses.

Findings

Randomized smoothing's theoretical robustness does not always translate to practical resilience.

Black-box attacks can often defeat randomized smoothing defenses under realistic settings.

There is a trade-off between achieving high certified robustness and maintaining classifier accuracy.

Abstract

Randomized smoothing is a recent and celebrated solution to certify the robustness of any classifier. While it indeed provides a theoretical robustness against adversarial attacks, the dimensionality of current classifiers necessarily imposes Monte Carlo approaches for its application in practice. This paper questions the effectiveness of randomized smoothing as a defense, against state of the art black-box attacks. This is a novel perspective, as previous research works considered the certification as an unquestionable guarantee. We first formally highlight the mismatch between a theoretical certification and the practice of attacks on classifiers. We then perform attacks on randomized smoothing as a defense. Our main observation is that there is a major mismatch in the settings of the RS for obtaining high certified robustness or when defeating black box attacks while preserving the classifier accuracy.