Neural Networks Perform Sufficient Dimension Reduction

TL;DR

This paper shows that neural networks naturally perform sufficient dimension reduction in regression tasks, with theoretical guarantees and empirical validation, highlighting their effectiveness in identifying key data subspaces.

Contribution

It establishes the connection between neural networks and SDR, proving neural networks can consistently estimate the central mean subspace under certain regularizations.

Findings

Neural networks span the central mean subspace in regression.

The estimator for the central mean subspace is statistically consistent.

Numerical experiments support the theoretical results.

Abstract

This paper investigates the connection between neural networks and sufficient dimension reduction (SDR), demonstrating that neural networks inherently perform SDR in regression tasks under appropriate rank regularizations. Specifically, the weights in the first layer span the central mean subspace. We establish the statistical consistency of the neural network-based estimator for the central mean subspace, underscoring the suitability of neural networks in addressing SDR-related challenges. Numerical experiments further validate our theoretical findings, and highlight the underlying capability of neural networks to facilitate SDR compared to the existing methods. Additionally, we discuss an extension to unravel the central subspace, broadening the scope of our investigation.