Adaptive Regularization for Large-Scale Sparse Feature Embedding Models

TL;DR

This paper investigates why large-scale sparse feature models overfit quickly during training, provides a theoretical explanation using Rademacher complexity, and proposes an adaptive regularization method that improves performance and prevents overfitting.

Contribution

It offers a theoretical understanding of overfitting in sparse models and introduces an adaptive regularization technique that enhances training stability and performance.

Findings

Theoretical explanation of overfitting using Rademacher complexity.

Adaptive regularization prevents performance degradation in multi-epoch training.

Method improves model performance within a single epoch and is deployed in production.

Abstract

The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, the fundamental cause of this phenomenon remains unclear. In this work, we present a theoretical explanation grounded in Rademacher complexity, supported by empirical experiments, to explain why overfitting occurs in models with large-scale sparse categorical features. Based on this analysis, we propose a regularization method that constrains the norm budget of embedding layers adaptively. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems.