Denoising Masked AutoEncoders Help Robust Classification

TL;DR

This paper introduces Denoising Masked AutoEncoders (DMAE), a self-supervised learning method that enhances the robustness of image classifiers by training on corrupted images, leading to state-of-the-art certified accuracy on ImageNet.

Contribution

The paper proposes DMAE, a novel self-supervised approach that improves certified robustness of classifiers and achieves superior results with fewer parameters.

Findings

DMAE achieves state-of-the-art certified accuracy on ImageNet.

Pre-trained DMAE models transfer well to CIFAR-10.

The method significantly outperforms previous robust classification techniques.

Abstract

In this paper, we propose a new self-supervised method, which is called Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers of images. In DMAE, we corrupt each image by adding Gaussian noises to each pixel value and randomly masking several patches. A Transformer-based encoder-decoder model is then trained to reconstruct the original image from the corrupted one. In this learning paradigm, the encoder will learn to capture relevant semantics for the downstream tasks, which is also robust to Gaussian additive noises. We show that the pre-trained encoder can naturally be used as the base classifier in Gaussian smoothed models, where we can analytically compute the certified radius for any data point. Although the proposed method is simple, it yields significant performance improvement in downstream classification tasks. We show that the DMAE ViT-Base model, which just uses 1/10 parameters of the model developed in recent work arXiv:2206.10550, achieves competitive or better certified accuracy in various settings. The DMAE ViT-Large model significantly surpasses all previous results, establishing a new state-of-the-art on ImageNet dataset. We further demonstrate that the pre-trained model has good transferability to the CIFAR-10 dataset, suggesting its wide adaptability. Models and code are available at https://github.com/quanlin-wu/dmae.