Deep Negative Correlation Classification

TL;DR

Deep Negative Correlation Classification (DNCC) introduces a novel ensemble method that optimizes both accuracy and diversity among models, improving efficiency and performance in deep learning tasks.

Contribution

DNCC systematically controls accuracy-diversity trade-off by decomposing the loss, enabling efficient, negatively correlated deep ensemble learning with shared backbones.

Findings

DNCC outperforms existing ensemble methods on benchmark datasets.

Shared backbone architecture maintains high accuracy with improved diversity.

Extensive experiments validate the effectiveness of DNCC.

Abstract

Ensemble learning serves as a straightforward way to improve the performance of almost any machine learning algorithm. Existing deep ensemble methods usually naively train many different models and then aggregate their predictions. This is not optimal in our view from two aspects: i) Naively training multiple models adds much more computational burden, especially in the deep learning era; ii) Purely optimizing each base model without considering their interactions limits the diversity of ensemble and performance gains. We tackle these issues by proposing deep negative correlation classification (DNCC), in which the accuracy and diversity trade-off is systematically controlled by decomposing the loss function seamlessly into individual accuracy and the correlation between individual models and the ensemble. DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated. Thanks to the optimized diversities, DNCC works well even when utilizing a shared network backbone, which significantly improves its efficiency when compared with most existing ensemble systems. Extensive experiments on multiple benchmark datasets and network structures demonstrate the superiority of the proposed method.