Exponential Machines

TL;DR

Exponential Machines (ExM) introduce a novel tensor factorization approach to model all feature interactions of any order, enabling efficient training and state-of-the-art performance in high-order interaction tasks.

Contribution

The paper presents Exponential Machines, a new model that uses Tensor Train factorization and Riemannian optimization to efficiently capture all feature interactions.

Findings

Achieves state-of-the-art results on synthetic high-order interaction data.

Performs comparably to high-order factorization machines on MovieLens 100K.

Successfully models tensors with up to 2^160 entries.

Abstract

Modeling interactions between features improves the performance of machine learning solutions in many domains (e.g. recommender systems or sentiment analysis). In this paper, we introduce Exponential Machines (ExM), a predictor that models all interactions of every order. The key idea is to represent an exponentially large tensor of parameters in a factorized format called Tensor Train (TT). The Tensor Train format regularizes the model and lets you control the number of underlying parameters. To train the model, we develop a stochastic Riemannian optimization procedure, which allows us to fit tensors with 2^160 entries. We show that the model achieves state-of-the-art performance on synthetic data with high-order interactions and that it works on par with high-order factorization machines on a recommender system dataset MovieLens 100K.