Exploring Resiliency to Natural Image Corruptions in Deep Learning using Design Diversity

TL;DR

This paper studies how diversity in deep learning model ensembles affects their robustness to natural image corruptions, highlighting the importance of architecture diversity and attribution-based metrics for improving resiliency.

Contribution

It introduces an attribution-based diversity metric and demonstrates that architecture diversity enhances ensemble resilience to image corruptions.

Findings

Model architecture impacts resiliency more than size or accuracy.

Attribution-based diversity correlates less negatively with accuracy.

Balanced loss functions improve ensemble robustness.

Abstract

In this paper, we investigate the relationship between diversity metrics, accuracy, and resiliency to natural image corruptions of Deep Learning (DL) image classifier ensembles. We investigate the potential of an attribution-based diversity metric to improve the known accuracy-diversity trade-off of the typical prediction-based diversity. Our motivation is based on analytical studies of design diversity that have shown that a reduction of common failure modes is possible if diversity of design choices is achieved. Using ResNet50 as a comparison baseline, we evaluate the resiliency of multiple individual DL model architectures against dataset distribution shifts corresponding to natural image corruptions. We compare ensembles created with diverse model architectures trained either independently or through a Neural Architecture Search technique and evaluate the correlation of prediction-based and attribution-based diversity to the final ensemble accuracy. We evaluate a set of diversity enforcement heuristics based on negative correlation learning to assess the final ensemble resilience to natural image corruptions and inspect the resulting prediction, activation, and attribution diversity. Our key observations are: 1) model architecture is more important for resiliency than model size or model accuracy, 2) attribution-based diversity is less negatively correlated to the ensemble accuracy than prediction-based diversity, 3) a balanced loss function of individual and ensemble accuracy creates more resilient ensembles for image natural corruptions, 4) architecture diversity produces more diversity in all explored diversity metrics: predictions, attributions, and activations.