Beyond Double Ascent via Recurrent Neural Tangent Kernel in Sequential Recommendation

TL;DR

This paper introduces OverRec, a method using Recurrent Neural Tangent Kernel to achieve large-model performance in sequential recommendation without training, revealing overfitting as a temporary phenomenon and demonstrating state-of-the-art results.

Contribution

It proposes OverRec, leveraging RNTK for recommendation, enabling large-scale model performance without training, and provides theoretical proof of RNTK's suitability for recommendation tasks.

Findings

OverRec achieves state-of-the-art results on four datasets.

Overfitting in large models is temporary and can be mitigated.

RNTK effectively captures user sequence similarities without training.

Abstract

Overfitting has long been considered a common issue to large neural network models in sequential recommendation. In our study, an interesting phenomenon is observed that overfitting is temporary. When the model scale is increased, the trend of the performance firstly ascends, then descends (i.e., overfitting) and finally ascends again, which is named as double ascent in this paper. We therefore raise an assumption that a considerably larger model will generalise better with a higher performance. In an extreme case to infinite-width, performance is expected to reach the limit of this specific structure. Unfortunately, it is impractical to directly build a huge model due to the limit of resources. In this paper, we propose the Overparameterised Recommender (OverRec), which utilises a recurrent neural tangent kernel (RNTK) as a similarity measurement for user sequences to successfully bypass the restriction of hardware for huge models. We further prove that the RNTK for the tied input-output embeddings in recommendation is the same as the RNTK for general untied input-output embeddings, which makes RNTK theoretically suitable for recommendation. Since the RNTK is analytically derived, OverRec does not require any training, avoiding physically building the huge model. Extensive experiments are conducted on four datasets, which verifies the state-of-the-art performance of OverRec.