Neural Factorization Machines for Sparse Predictive Analytics

TL;DR

This paper introduces Neural Factorization Machines (NFM), a model that combines the linear modeling of feature interactions in FMs with neural networks to capture complex, non-linear relationships in sparse data, outperforming existing methods.

Contribution

The paper proposes NFM, a novel model that integrates FM's linear second-order interactions with neural networks for higher-order interactions, improving predictive performance and training simplicity.

Findings

NFM outperforms FM with a 7.3% relative improvement on regression tasks.

NFM surpasses Wide&Deep and DeepCross with a shallower, easier-to-train structure.

Empirical results demonstrate NFM's superior ability to model complex feature interactions.

Abstract

Many predictive tasks of web applications need to model categorical variables, such as user IDs and demographics like genders and occupations. To apply standard machine learning techniques, these categorical predictors are always converted to a set of binary features via one-hot encoding, making the resultant feature vector highly sparse. To learn from such sparse data effectively, it is crucial to account for the interactions between features. Factorization Machines (FMs) are a popular solution for efficiently using the second-order feature interactions. However, FM models feature interactions in a linear way, which can be insufficient for capturing the non-linear and complex inherent structure of real-world data. While deep neural networks have recently been applied to learn non-linear feature interactions in industry, such as the Wide&Deep by Google and DeepCross by Microsoft, the deep structure meanwhile makes them difficult to train. In this paper, we propose a novel model Neural Factorization Machine (NFM) for prediction under sparse settings. NFM seamlessly combines the linearity of FM in modelling second-order feature interactions and the non-linearity of neural network in modelling higher-order feature interactions. Conceptually, NFM is more expressive than FM since FM can be seen as a special case of NFM without hidden layers. Empirical results on two regression tasks show that with one hidden layer only, NFM significantly outperforms FM with a 7.3% relative improvement. Compared to the recent deep learning methods Wide&Deep and DeepCross, our NFM uses a shallower structure but offers better performance, being much easier to train and tune in practice.