Gradient Boosting Mapping for Dimensionality Reduction and Feature Extraction

TL;DR

This paper introduces Gradient Boosting Mapping (GBMAP), a supervised dimensionality reduction technique that creates effective features and distance measures, improving model interpretability, reducing overfitting, and detecting out-of-distribution data efficiently.

Contribution

GBMAP is a novel supervised dimensionality reduction method using weak learners to generate embeddings that enhance learning performance and enable principled distance measurement.

Findings

GBMAP produces features that improve supervised learning performance.

It enables detection of out-of-distribution data points.

The method is fast, scalable, and competitive with state-of-the-art models.

Abstract

A fundamental problem in supervised learning is to find a good set of features or distance measures. If the new set of features is of lower dimensionality and can be obtained by a simple transformation of the original data, they can make the model understandable, reduce overfitting, and even help to detect distribution drift. We propose a supervised dimensionality reduction method Gradient Boosting Mapping (GBMAP), where the outputs of weak learners -- defined as one-layer perceptrons -- define the embedding. We show that the embedding coordinates provide better features for the supervised learning task, making simple linear models competitive with the state-of-the-art regressors and classifiers. We also use the embedding to find a principled distance measure between points. The features and distance measures automatically ignore directions irrelevant to the supervised learning task. We also show that we can reliably detect out-of-distribution data points with potentially large regression or classification errors. GBMAP is fast and works in seconds for dataset of million data points or hundreds of features. As a bonus, GBMAP provides a regression and classification performance comparable to the state-of-the-art supervised learning methods.