Persistent Reductions in Regularized Loss Minimization for Variable Selection

TL;DR

This paper introduces a data-driven, pre-optimization feature reduction method for regularized loss minimization problems with polyhedral gauges, capable of identifying zero-coefficient features across all solutions efficiently.

Contribution

The authors propose a novel persistent reduction technique that pre-identifies zero-coefficient features without iterative optimization, applicable to a broad class of loss functions and high-dimensional data.

Findings

The reduction guarantees feature elimination before optimization.

The method is data-only, requiring no loss function calls.

It is efficient in ultra-high dimensional settings.

Abstract

In the context of regularized loss minimization with polyhedral gauges, we show that for a broad class of loss functions (possibly non-smooth and non-convex) and under a simple geometric condition on the input data it is possible to efficiently identify a subset of features which are guaranteed to have zero coefficients in all optimal solutions in all problems with loss functions from said class, before any iterative optimization has been performed for the original problem. This procedure is standalone, takes only the data as input, and does not require any calls to the loss function. Therefore, we term this procedure as a persistent reduction for the aforementioned class of regularized loss minimization problems. This reduction can be efficiently implemented via an extreme ray identification subroutine applied to a polyhedral cone formed from the datapoints. We employ an existing output-sensitive algorithm for extreme ray identification which makes our guarantee and algorithm applicable in ultra-high dimensional problems.