DIFER: Differentiable Automated Feature Engineering

TL;DR

DIFER introduces a gradient-based, differentiable approach to automated feature engineering that efficiently optimizes features in a continuous space, significantly enhancing machine learning performance over existing methods.

Contribution

The paper presents DIFER, a novel differentiable AutoFE method that uses continuous embeddings and gradient optimization to improve feature selection efficiency and effectiveness.

Findings

DIFER outperforms state-of-the-art AutoFE methods in accuracy.

DIFER reduces computational cost compared to black-box optimization approaches.

DIFER enhances performance across various machine learning tasks.

Abstract

Feature engineering, a crucial step of machine learning, aims to extract useful features from raw data to improve data quality. In recent years, great efforts have been devoted to Automated Feature Engineering (AutoFE) to replace expensive human labor. However, existing methods are computationally demanding due to treating AutoFE as a coarse-grained black-box optimization problem over a discrete space. In this work, we propose an efficient gradient-based method called DIFER to perform differentiable automated feature engineering in a continuous vector space. DIFER selects potential features based on evolutionary algorithm and leverages an encoder-predictor-decoder controller to optimize existing features. We map features into the continuous vector space via the encoder, optimize the embedding along the gradient direction induced by the predicted score, and recover better features from the optimized embedding by the decoder. Extensive experiments on classification and regression datasets demonstrate that DIFER can significantly improve the performance of various machine learning algorithms and outperform current state-of-the-art AutoFE methods in terms of both efficiency and performance.