Double Sparsity Kernel Learning with Automatic Variable Selection and Data Extraction

TL;DR

This paper introduces DOSK, a novel RKHS learning method that simultaneously performs variable selection and data extraction, addressing limitations of traditional approaches and demonstrating strong theoretical and empirical performance.

Contribution

The paper proposes a unified RKHS learning framework called DOSK that integrates variable selection and data extraction, with an efficient algorithm and theoretical guarantees.

Findings

DOSK achieves variable selection consistency asymptotically.

DOSK outperforms existing methods in simulated and real data experiments.

The method effectively handles noise predictors and sparse representations.

Abstract

Learning with Reproducing Kernel Hilbert Spaces (RKHS) has been widely used in many scientific disciplines. Because a RKHS can be very flexible, it is common to impose a regularization term in the optimization to prevent overfitting. Standard RKHS learning employs the squared norm penalty of the learning function. Despite its success, many challenges remain. In particular, one cannot directly use the squared norm penalty for variable selection or data extraction. Therefore, when there exists noise predictors, or the underlying function has a sparse representation in the dual space, the performance of standard RKHS learning can be suboptimal. In the literature,work has been proposed on how to perform variable selection in RKHS learning, and a data sparsity constraint was considered for data extraction. However, how to learn in a RKHS with both variable selection and data extraction simultaneously remains unclear. In this paper, we propose a unified RKHS learning method, namely, DOuble Sparsity Kernel (DOSK) learning, to overcome this challenge. An efficient algorithm is provided to solve the corresponding optimization problem. We prove that under certain conditions, our new method can asymptotically achieve variable selection consistency. Simulated and real data results demonstrate that DOSK is highly competitive among existing approaches for RKHS learning.