Kernels and Ensembles: Perspectives on Statistical Learning

TL;DR

This paper reviews kernel and ensemble methods in statistical learning, highlighting their core ideas, personal perspectives, and recent innovations like LAGO and Darwinian evolution algorithms.

Contribution

It provides an expository overview of kernel and ensemble methods and introduces two novel algorithms developed by the author and collaborators.

Findings

LAGO is a fast kernel algorithm for unbalanced classification.

Ensemble method using Darwinian evolution for variable selection.

Personal insights on the impact of these methods on research.

Abstract

Since their emergence in the 1990's, the support vector machine and the AdaBoost algorithm have spawned a wave of research in statistical machine learning. Much of this new research falls into one of two broad categories: kernel methods and ensemble methods. In this expository article, I discuss the main ideas behind these two types of methods, namely how to transform linear algorithms into nonlinear ones by using kernel functions, and how to make predictions with an ensemble or a collection of models rather than a single model. I also share my personal perspectives on how these ideas have influenced and shaped my own research. In particular, I present two recent algorithms that I have invented with my collaborators: LAGO, a fast kernel algorithm for unbalanced classification and rare target detection; and Darwinian evolution in parallel universes, an ensemble method for variable selection.