Learning Robust Kernel Ensembles with Kernel Average Pooling

TL;DR

This paper introduces Kernel Average Pooling (KAP), a novel neural network component that enhances robustness against adversarial attacks by promoting kernel ensembles, demonstrated through extensive experiments on multiple datasets.

Contribution

The paper proposes Kernel Average Pooling (KAP), a new building block that naturally encourages kernel ensembles and improves neural network robustness without adversarial training.

Findings

KAP models are highly robust against adversarial attacks.

KAP improves robustness on CIFAR and ImageNet datasets.

KAP enhances model stability without adversarial example training.

Abstract

Model ensembles have long been used in machine learning to reduce the variance in individual model predictions, making them more robust to input perturbations. Pseudo-ensemble methods like dropout have also been commonly used in deep learning models to improve generalization. However, the application of these techniques to improve neural networks' robustness against input perturbations remains underexplored. We introduce Kernel Average Pooling (KAP), a neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor. We show that ensembles of kernels with similar functionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation. Moreover, we show that when trained on inputs perturbed with additive Gaussian noise, KAP models are remarkably robust against various forms of adversarial attacks. Empirical evaluations on CIFAR10, CIFAR100, TinyImagenet, and Imagenet datasets show substantial improvements in robustness against strong adversarial attacks such as AutoAttack without training on any adversarial examples.