On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective

TL;DR

This paper reveals that weight decay in collaborative filtering encodes item popularity into embeddings, and introduces PRISM, a method that pre-encodes popularity to improve training efficiency and performance.

Contribution

The work uncovers the role of weight decay in encoding popularity and proposes PRISM, a new initialization strategy that eliminates the need for weight decay in CF models.

Findings

PRISM improves recommendation accuracy by up to 4.77%.

PRISM reduces training time by 38.48%.

Tuning weight decay influences preference towards popular or unpopular items.

Abstract

Collaborative filtering (CF) enables large-scale recommendation systems by encoding information from historical user-item interactions into dense ID-embedding tables. However, as embedding tables grow, closed-form solutions become impractical, often necessitating the use of mini-batch gradient descent for training. Despite extensive work on designing loss functions to train CF models, we argue that one core component of these pipelines is heavily overlooked: weight decay. Attaining high-performing models typically requires careful tuning of weight decay, regardless of loss, yet its necessity is not well understood. In this work, we question why weight decay is crucial in CF pipelines and how it impacts training. Through theoretical and empirical analysis, we surprisingly uncover that weight decay's primary function is to encode popularity information into the magnitudes of the embedding vectors. Moreover, we find that tuning weight decay acts as a coarse, non-linear knob to influence preference towards popular or unpopular items. Based on these findings, we propose PRISM (Popularity-awaRe Initialization Strategy for embedding Magnitudes), a straightforward yet effective solution to simplify the training of high-performing CF models. PRISM pre-encodes the popularity information typically learned through weight decay, eliminating its necessity. Our experiments show that PRISM improves performance by up to 4.77% and reduces training times by 38.48%, compared to state-of-the-art training strategies. Additionally, we parameterize PRISM to modulate the initialization strength, offering a cost-effective and meaningful strategy to mitigate popularity bias.