Efficient Learning of Sparse Representations from Interactions

TL;DR

This paper introduces a training method for high-dimensional sparse embeddings that significantly reduces size and maintains accuracy, improving efficiency and interpretability in recommender systems.

Contribution

The authors develop a strategy for learning sparse embeddings that are both compact and expressive, demonstrated by modifying ELSA to achieve substantial size reductions with minimal accuracy loss.

Findings

Up to 10x reduction in embedding size without accuracy loss

Up to 100x reduction with only 2.5% accuracy loss

Embedding dimensions reveal interpretable item segments

Abstract

Behavioral patterns captured in embeddings learned from interaction data are pivotal across various stages of production recommender systems. However, in the initial retrieval stage, practitioners face an inherent tradeoff between embedding expressiveness and the scalability and latency of serving components, resulting in the need for representations that are both compact and expressive. To address this challenge, we propose a training strategy for learning high-dimensional sparse embedding layers in place of conventional dense ones, balancing efficiency, representational expressiveness, and interpretability. To demonstrate our approach, we modified the production-grade collaborative filtering autoencoder ELSA, achieving up to 10x reduction in embedding size with no loss of recommendation accuracy, and up to 100x reduction with only a 2.5% loss. Moreover, the active embedding dimensions reveal an interpretable inverted-index structure that segments items in a way directly aligned with the model's latent space, thereby enabling integration of segment-level recommendation functionality (e.g., 2D homepage layouts) within the candidate retrieval model itself. Source codes, additional results, as well as a live demo are available at https://github.com/zombak79/compressed_elsa