Safe RuleFit: Learning Optimal Sparse Rule Model by Meta Safe Screening

TL;DR

Safe RuleFit introduces a novel meta safe screening technique to efficiently learn sparse rule-based models, significantly reducing computational complexity while maintaining optimality in regression and classification tasks.

Contribution

The paper proposes Safe RuleFit with meta safe screening, enabling efficient selection of rules in large rule spaces for sparse models, extending to various regularizations.

Findings

Effective screening of multiple rules reduces computational cost.

Framework applicable to regression and classification tasks.

Demonstrated advantages through extensive experiments.

Abstract

We consider the problem of learning a sparse rule model, a prediction model in the form of a sparse linear combination of rules, where a rule is an indicator function defined over a hyper-rectangle in the input space. Since the number of all possible such rules is extremely large, it has been computationally intractable to select the optimal set of active rules. In this paper, to solve this difficulty for learning the optimal sparse rule model, we propose Safe RuleFit (SRF). Our basic idea is to develop meta safe screening (mSS), which is a non-trivial extension of well-known safe screening (SS) techniques. While SS is used for screening out one feature, mSS can be used for screening out multiple features by exploiting the inclusion-relations of hyper-rectangles in the input space. SRF provides a general framework for fitting sparse rule models for regression and classification, and it can be extended to handle more general sparse regularizations such as group regularization. We demonstrate the advantages of SRF through intensive numerical experiments.