Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations

TL;DR

This paper addresses the reproducibility issues in large-scale recommendation systems driven by deep networks, identifying ReLU as a key factor, and proposes smooth activations like SmeLU to enhance reproducibility and accuracy.

Contribution

It introduces SmeLU, a novel smooth activation function, demonstrating its effectiveness in improving reproducibility and accuracy in real-world recommendation systems.

Findings

SmeLU improves reproducibility in CTR prediction systems.

Smooth activations can enhance both accuracy and reproducibility.

ReLU significantly contributes to irreproducibility in deep recommendation models.

Abstract

Real world recommendation systems influence a constantly growing set of domains. With deep networks, that now drive such systems, recommendations have been more relevant to the user's interests and tasks. However, they may not always be reproducible even if produced by the same system for the same user, recommendation sequence, request, or query. This problem received almost no attention in academic publications, but is, in fact, very realistic and critical in real production systems. We consider reproducibility of real large scale deep models, whose predictions determine such recommendations. We demonstrate that the celebrated Rectified Linear Unit (ReLU) activation, used in deep models, can be a major contributor to irreproducibility. We propose the use of smooth activations to improve recommendation reproducibility. We describe a novel family of smooth activations; Smooth ReLU (SmeLU), designed to improve reproducibility with mathematical simplicity, with potentially cheaper implementation. SmeLU is a member of a wider family of smooth activations. While other techniques that improve reproducibility in real systems usually come at accuracy costs, smooth activations not only improve reproducibility, but can even give accuracy gains. We report metrics from real systems in which we were able to productionalize SmeLU with substantial reproducibility gains and better accuracy-reproducibility trade-offs. These include click-through-rate (CTR) prediction systems, content, and application recommendation systems.