Learning Robust Variational Information Bottleneck with Reference

TL;DR

This paper introduces a robust variational information bottleneck method that leverages soft labels from a reference network and mutual information neural estimation to enhance adversarial robustness, demonstrating superior performance on MNIST and CIFAR-10.

Contribution

It presents a novel VIB training approach using soft labels and relaxed Gaussian assumptions, improving robustness against adversarial attacks.

Findings

Significant robustness improvement over benchmarks.

Effective use of soft labels from reference networks.

Successful application on MNIST and CIFAR-10 datasets.

Abstract

We propose a new approach to train a variational information bottleneck (VIB) that improves its robustness to adversarial perturbations. Unlike the traditional methods where the hard labels are usually used for the classification task, we refine the categorical class information in the training phase with soft labels which are obtained from a pre-trained reference neural network and can reflect the likelihood of the original class labels. We also relax the Gaussian posterior assumption in the VIB implementation by using the mutual information neural estimation. Extensive experiments have been performed with the MNIST and CIFAR-10 datasets, and the results show that our proposed approach significantly outperforms the benchmarked models.