Towards Ultrahigh Dimensional Feature Selection for Big Data

TL;DR

This paper introduces a novel adaptive feature scaling method for ultrahigh-dimensional feature selection in Big Data, reformulating the problem as a convex SIP and employing a feature generating paradigm for efficiency and effectiveness.

Contribution

It proposes a new feature generating paradigm with a primal MKL approach, enabling scalable, nonlinear, and group-based feature selection with guaranteed convergence and lower bias.

Findings

Outperforms state-of-the-art methods in accuracy and efficiency.

Successfully handles datasets with tens of millions of data points and trillions of features.

Demonstrates effectiveness in both synthetic and real-world scenarios.

Abstract

In this paper, we present a new adaptive feature scaling scheme for ultrahigh-dimensional feature selection on Big Data. To solve this problem effectively, we first reformulate it as a convex semi-infinite programming (SIP) problem and then propose an efficient \emph{feature generating paradigm}. In contrast with traditional gradient-based approaches that conduct optimization on all input features, the proposed method iteratively activates a group of features and solves a sequence of multiple kernel learning (MKL) subproblems of much reduced scale. To further speed up the training, we propose to solve the MKL subproblems in their primal forms through a modified accelerated proximal gradient approach. Due to such an optimization scheme, some efficient cache techniques are also developed. The feature generating paradigm can guarantee that the solution converges globally under mild conditions and achieve lower feature selection bias. Moreover, the proposed method can tackle two challenging tasks in feature selection: 1) group-based feature selection with complex structures and 2) nonlinear feature selection with explicit feature mappings. Comprehensive experiments on a wide range of synthetic and real-world datasets containing tens of million data points with $O(10^{14})$ features demonstrate the competitive performance of the proposed method over state-of-the-art feature selection methods in terms of generalization performance and training efficiency.