Memory-Efficient Factorization Machines via Binarizing both Data and Model Coefficients

TL;DR

This paper introduces Binarized FM, a memory-efficient variant of SEFM that constrains model parameters to binary values, significantly reducing memory usage while maintaining comparable accuracy across multiple datasets.

Contribution

The paper proposes Binarized FM, a novel method that reduces memory costs of SEFM by binarizing model parameters and introduces an effective learning algorithm using STE and Adagrad.

Findings

Binarized FM achieves similar accuracy to SEFM.

Memory usage is significantly reduced with binarization.

Effective training with STE and Adagrad.

Abstract

Factorization Machines (FM), a general predictor that can efficiently model feature interactions in linear time, was primarily proposed for collaborative recommendation and have been broadly used for regression, classification and ranking tasks. Subspace Encoding Factorization Machine (SEFM) has been proposed recently to overcome the expressiveness limitation of Factorization Machines (FM) by applying explicit nonlinear feature mapping for both individual features and feature interactions through one-hot encoding to each input feature. Despite the effectiveness of SEFM, it increases the memory cost of FM by $b$ times, where $b$ is the number of bins when applying one-hot encoding on each input feature. To reduce the memory cost of SEFM, we propose a new method called Binarized FM which constraints the model parameters to be binary values (i.e., 1 or $-1$). Then each parameter value can be efficiently stored in one bit. Our proposed method can significantly reduce the memory cost of SEFM model. In addition, we propose a new algorithm to effectively and efficiently learn proposed FM with binary constraints using Straight Through Estimator (STE) with Adaptive Gradient Descent (Adagrad). Finally, we evaluate the performance of our proposed method on eight different classification datasets. Our experimental results have demonstrated that our proposed method achieves comparable accuracy with SEFM but with much less memory cost.