Model-Agnostic and Uncertainty-Aware Dimensionality Reduction in Supervised Learning

TL;DR

This paper presents POD, a flexible, model-agnostic framework for dimension reduction in supervised learning that directly assesses predictive utility and quantifies uncertainty, improving the selection of intrinsic data dimensions.

Contribution

Introduces POD, a novel, predictive performance-based, model-agnostic approach for determining the minimal sufficient data dimension with uncertainty quantification.

Findings

POD accurately estimates data dimensions in simulations.

POD provides reliable uncertainty bounds for order estimation.

POD outperforms traditional methods in real-data applications.

Abstract

Dimension reduction is a fundamental tool for analyzing high-dimensional data in supervised learning. Traditional methods for estimating intrinsic order often prioritize model-specific structural assumptions over predictive utility. This paper introduces predictive order determination (POD), a model-agnostic framework that determines the minimal predictively sufficient dimension by directly evaluating out-of-sample predictiveness. POD quantifies uncertainty via error bounds for over- and underestimation and achieves consistency under mild conditions. By unifying dimension reduction with predictive performance, POD applies flexibly across diverse reduction tasks and supervised learners. Simulations and real-data analyses show that POD delivers accurate, uncertainty-aware order estimates, making it a versatile component for prediction-centric pipelines.