Learning Better Certified Models from Empirically-Robust Teachers

TL;DR

This paper introduces a method to enhance certifiably robust neural networks by distilling knowledge from empirically-robust teachers, achieving better certified robustness without sacrificing standard accuracy.

Contribution

It proposes a novel feature-space distillation approach from adversarially-trained teachers to improve certified robustness in neural networks.

Findings

Distillation from robust teachers improves certified robustness.

Achieves state-of-the-art results on robust benchmarks.

Enhances the trade-off between robustness and accuracy.

Abstract

Adversarial training attains strong empirical robustness to specific adversarial attacks by training on concrete adversarial perturbations, but it produces neural networks that are not amenable to strong robustness certificates through neural network verification. On the other hand, earlier certified training schemes directly train on bounds from network relaxations to obtain models that are certifiably robust, but display sub-par standard performance. Recent work has shown that state-of-the-art trade-offs between certified robustness and standard performance can be obtained through a family of losses combining adversarial outputs and neural network bounds. Nevertheless, differently from empirical robustness, verifiability still comes at a significant cost in standard performance. In this work, we propose to leverage empirically-robust teachers to improve the performance of certifiably-robust models through knowledge distillation. Using a versatile feature-space distillation objective, we show that distillation from adversarially-trained teachers consistently improves on the state-of-the-art in certified training for ReLU networks across a series of robust computer vision benchmarks.