Feature-Weighted Linear Stacking

TL;DR

Feature-Weighted Linear Stacking (FWLS) is a linear ensemble method that uses meta-features to improve predictive accuracy efficiently, demonstrated on the Netflix Prize dataset.

Contribution

The paper introduces FWLS, a linear stacking technique that incorporates meta-features for enhanced accuracy while maintaining simplicity and interpretability.

Findings

FWLS outperforms standard linear stacking in accuracy.

Demonstrated significant improvements on Netflix Prize data.

FWLS was a key component in a top-performing Netflix Prize solution.

Abstract

Ensemble methods, such as stacking, are designed to boost predictive accuracy by blending the predictions of multiple machine learning models. Recent work has shown that the use of meta-features, additional inputs describing each example in a dataset, can boost the performance of ensemble methods, but the greatest reported gains have come from nonlinear procedures requiring significant tuning and training time. Here, we present a linear technique, Feature-Weighted Linear Stacking (FWLS), that incorporates meta-features for improved accuracy while retaining the well-known virtues of linear regression regarding speed, stability, and interpretability. FWLS combines model predictions linearly using coefficients that are themselves linear functions of meta-features. This technique was a key facet of the solution of the second place team in the recently concluded Netflix Prize competition. Significant increases in accuracy over standard linear stacking are demonstrated on the Netflix Prize collaborative filtering dataset.