Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement

TL;DR

This paper introduces the Amended Cross Entropy (ACE), a new cost function that enables training multiple classifiers with explicit control over their diversity, leading to improved ensemble performance.

Contribution

It proposes a novel cost function, ACE, that explicitly encourages diversity among classifiers, enhancing ensemble learning effectiveness.

Findings

ACE outperforms vanilla ensembles in experiments.

Optimal diversity factor improves ensemble accuracy.

Method applicable to classification similarly to NCL for regression.

Abstract

Cross Entropy (CE) has an important role in machine learning and, in particular, in neural networks. It is commonly used in neural networks as the cost between the known distribution of the label and the Softmax/Sigmoid output. In this paper we present a new cost function called the Amended Cross Entropy (ACE). Its novelty lies in its affording the capability to train multiple classifiers while explicitly controlling the diversity between them. We derived the new cost by mathematical analysis and "reverse engineering" of the way we wish the gradients to behave, and produced a tailor-made, elegant and intuitive cost function to achieve the desired result. This process is similar to the way that CE cost is picked as a cost function for the Softmax/Sigmoid classifiers for obtaining linear derivatives. By choosing the optimal diversity factor we produce an ensemble which yields better results than the vanilla one. We demonstrate two potential usages of this outcome, and present empirical results. Our method works for classification problems analogously to Negative Correlation Learning (NCL) for regression problems.