Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness

TL;DR

This paper demonstrates that heterogeneous deep neural network ensembles, constructed through diversity-promoting methods, can significantly improve robustness against negative examples and adversarial attacks, supported by formal analysis and extensive experiments.

Contribution

It introduces a novel two-tier heterogeneity-driven ensemble construction method that enhances robustness by promoting diversity and reducing negative correlation among models.

Findings

Heterogeneous models improve mean average precision in object detection.

Ensembles of different tasks (detection and segmentation) increase robustness.

Formal analysis links negative correlation to ensemble robustness.

Abstract

Deep neural network ensembles hold the potential of improving generalization performance for complex learning tasks. This paper presents formal analysis and empirical evaluation to show that heterogeneous deep ensembles with high ensemble diversity can effectively leverage model learning heterogeneity to boost ensemble robustness. We first show that heterogeneous DNN models trained for solving the same learning problem, e.g., object detection, can significantly strengthen the mean average precision (mAP) through our weighted bounding box ensemble consensus method. Second, we further compose ensembles of heterogeneous models for solving different learning problems, e.g., object detection and semantic segmentation, by introducing the connected component labeling (CCL) based alignment. We show that this two-tier heterogeneity driven ensemble construction method can compose an ensemble team that promotes high ensemble diversity and low negative correlation among member models of the ensemble, strengthening ensemble robustness against both negative examples and adversarial attacks. Third, we provide a formal analysis of the ensemble robustness in terms of negative correlation. Extensive experiments validate the enhanced robustness of heterogeneous ensembles in both benign and adversarial settings. The source codes are available on GitHub at https://github.com/git-disl/HeteRobust.