Implicit Regularization for Multi-label Feature Selection

TL;DR

This paper introduces a novel implicit regularization approach for multi-label feature selection that leverages label embedding and Hadamard product parameterization, reducing bias compared to traditional penalized methods.

Contribution

The paper proposes a simple, bias-reducing feature selection method using implicit regularization and label embedding, differing from traditional explicit regularization techniques.

Findings

Less bias in feature selection compared to penalized methods

Potential for benign overfitting in multi-label learning

Effective on benchmark datasets

Abstract

In this paper, we address the problem of feature selection in the context of multi-label learning, by using a new estimator based on implicit regularization and label embedding. Unlike the sparse feature selection methods that use a penalized estimator with explicit regularization terms such as $l_{2,1}$-norm, MCP or SCAD, we propose a simple alternative method via Hadamard product parameterization. In order to guide the feature selection process, a latent semantic of multi-label information method is adopted, as a label embedding. Experimental results on some known benchmark datasets suggest that the proposed estimator suffers much less from extra bias, and may lead to benign overfitting.