RULLS: Randomized Union of Locally Linear Subspaces for Feature Engineering

TL;DR

RULLS is a novel feature engineering technique that creates robust, sparse, and rotation-invariant features by aggregating local subspace information around randomly selected landmarks, improving clustering and classification performance.

Contribution

The paper introduces RULLS, a new unsupervised feature engineering method that generates meaningful features using a union of locally linear subspaces based on randomly chosen landmarks.

Findings

RULLS outperforms existing feature generation methods in clustering tasks.

RULLS achieves high classification accuracy on real-world datasets.

The method is robust to noise and effective in various data scenarios.

Abstract

Feature engineering plays an important role in the success of a machine learning model. Most of the effort in training a model goes into data preparation and choosing the right representation. In this paper, we propose a robust feature engineering method, Randomized Union of Locally Linear Subspaces (RULLS). We generate sparse, non-negative, and rotation invariant features in an unsupervised fashion. RULLS aggregates features from a random union of subspaces by describing each point using globally chosen landmarks. These landmarks serve as anchor points for choosing subspaces. Our method provides a way to select features that are relevant in the neighborhood around these chosen landmarks. Distances from each data point to $k$ closest landmarks are encoded in the feature matrix. The final feature representation is a union of features from all chosen subspaces. The effectiveness of our algorithm is shown on various real-world datasets for tasks such as clustering and classification of raw data and in the presence of noise. We compare our method with existing feature generation methods. Results show a high performance of our method on both classification and clustering tasks.