Human uncertainty makes classification more robust

TL;DR

This paper demonstrates that training deep neural networks with human perceptual uncertainty data improves their generalization and robustness to adversarial attacks, addressing limitations of current models.

Contribution

The introduction of CIFAR10H, a new dataset with human label distributions, and evidence that training on it enhances model robustness and out-of-distribution generalization.

Findings

Training with human uncertainty improves robustness to adversarial attacks.

Models trained on CIFAR10H better generalize to out-of-distribution data.

Explicit modeling of human perceptual uncertainty benefits deep learning classifiers.

Abstract

The classification performance of deep neural networks has begun to asymptote at near-perfect levels. However, their ability to generalize outside the training set and their robustness to adversarial attacks have not. In this paper, we make progress on this problem by training with full label distributions that reflect human perceptual uncertainty. We first present a new benchmark dataset which we call CIFAR10H, containing a full distribution of human labels for each image of the CIFAR10 test set. We then show that, while contemporary classifiers fail to exhibit human-like uncertainty on their own, explicit training on our dataset closes this gap, supports improved generalization to increasingly out-of-training-distribution test datasets, and confers robustness to adversarial attacks.