Sliding Spectrum Decomposition for Diversified Recommendation

TL;DR

This paper introduces sliding spectrum decomposition (SSD), a novel time series analysis technique for enhancing diversity in content feed recommendations, successfully deployed in a large-scale social media app.

Contribution

The paper presents SSD, a new method for measuring diversity in item sequences, along with an effective item embedding approach, both deployed in a real-world recommender system.

Findings

SSD effectively captures user diversity perception.

The method improves recommendation diversity in offline and online tests.

Deployment in Xiaohongshu's system benefits millions of users daily.

Abstract

Content feed, a type of product that recommends a sequence of items for users to browse and engage with, has gained tremendous popularity among social media platforms. In this paper, we propose to study the diversity problem in such a scenario from an item sequence perspective using time series analysis techniques. We derive a method called sliding spectrum decomposition (SSD) that captures users' perception of diversity in browsing a long item sequence. We also share our experiences in designing and implementing a suitable item embedding method for accurate similarity measurement under long tail effect. Combined together, they are now fully implemented and deployed in Xiaohongshu App's production recommender system that serves the main Explore Feed product for tens of millions of users every day. We demonstrate the effectiveness and efficiency of the method through theoretical analysis, offline experiments and online A/B tests.