Ensembles of Classifiers based on Dimensionality Reduction

TL;DR

This paper introduces a new ensemble classification method using various dimensionality reduction techniques to improve accuracy and diversity, demonstrating superior results on benchmark datasets.

Contribution

It proposes a novel ensemble approach based on multiple dimensionality reduction methods and combines strategies like AdaBoost, outperforming traditional ensemble methods.

Findings

Proposed methods outperform traditional ensembles on several datasets.

Dimensionality reduction enhances ensemble diversity and accuracy.

Multi-strategy ensemble with AdaBoost and Diffusion Maps shows improved performance.

Abstract

We present a novel approach for the construction of ensemble classifiers based on dimensionality reduction. Dimensionality reduction methods represent datasets using a small number of attributes while preserving the information conveyed by the original dataset. The ensemble members are trained based on dimension-reduced versions of the training set. These versions are obtained by applying dimensionality reduction to the original training set using different values of the input parameters. This construction meets both the diversity and accuracy criteria which are required to construct an ensemble classifier where the former criterion is obtained by the various input parameter values and the latter is achieved due to the decorrelation and noise reduction properties of dimensionality reduction. In order to classify a test sample, it is first embedded into the dimension reduced space of each individual classifier by using an out-of-sample extension algorithm. Each classifier is then applied to the embedded sample and the classification is obtained via a voting scheme. We present three variations of the proposed approach based on the Random Projections, the Diffusion Maps and the Random Subspaces dimensionality reduction algorithms. We also present a multi-strategy ensemble which combines AdaBoost and Diffusion Maps. A comparison is made with the Bagging, AdaBoost, Rotation Forest ensemble classifiers and also with the base classifier which does not incorporate dimensionality reduction. Our experiments used seventeen benchmark datasets from the UCI repository. The results obtained by the proposed algorithms were superior in many cases to other algorithms.