Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey

TL;DR

This tutorial and survey comprehensively review various sufficient dimension reduction methods for high-dimensional regression, covering inverse and forward regression techniques, as well as kernel methods, highlighting their statistical and machine learning perspectives.

Contribution

It provides a unified overview of SDR methods, including recent deep learning approaches, and clarifies the relationship between supervised KDR and PCA.

Findings

Survey of classical and modern SDR methods

Equivalence between supervised KDR and PCA

Comparison of inverse and forward regression approaches

Abstract

This is a tutorial and survey paper on various methods for Sufficient Dimension Reduction (SDR). We cover these methods with both statistical high-dimensional regression perspective and machine learning approach for dimensionality reduction. We start with introducing inverse regression methods including Sliced Inverse Regression (SIR), Sliced Average Variance Estimation (SAVE), contour regression, directional regression, Principal Fitted Components (PFC), Likelihood Acquired Direction (LAD), and graphical regression. Then, we introduce forward regression methods including Principal Hessian Directions (pHd), Minimum Average Variance Estimation (MAVE), Conditional Variance Estimation (CVE), and deep SDR methods. Finally, we explain Kernel Dimension Reduction (KDR) both for supervised and unsupervised learning. We also show that supervised KDR and supervised PCA are equivalent.