Statistical Test for Auto Feature Engineering by Selective Inference

TL;DR

This paper introduces a statistical test based on selective inference to evaluate the reliability of features generated by auto feature engineering algorithms, providing a way to control false discoveries.

Contribution

It proposes a novel statistical testing framework for features from AFE algorithms, enabling significance assessment with theoretical guarantees.

Findings

The test quantifies feature significance with p-values.

It offers controlled false discovery risk.

Applicable to tree search-based AFE algorithms.

Abstract

Auto Feature Engineering (AFE) plays a crucial role in developing practical machine learning pipelines by automating the transformation of raw data into meaningful features that enhance model performance. By generating features in a data-driven manner, AFE enables the discovery of important features that may not be apparent through human experience or intuition. On the other hand, since AFE generates features based on data, there is a risk that these features may be overly adapted to the data, making it essential to assess their reliability appropriately. Unfortunately, because most AFE problems are formulated as combinatorial search problems and solved by heuristic algorithms, it has been challenging to theoretically quantify the reliability of generated features. To address this issue, we propose a new statistical test for generated features by AFE algorithms based on a framework called selective inference. As a proof of concept, we consider a simple class of tree search-based heuristic AFE algorithms, and consider the problem of testing the generated features when they are used in a linear model. The proposed test can quantify the statistical significance of the generated features in the form of $p$-values, enabling theoretically guaranteed control of the risk of false findings.