Efficient Retrieval Scaling with Hierarchical Indexing for Large Scale Recommendation

TL;DR

This paper introduces a hierarchical indexing method for large-scale retrieval models, enabling more efficient search while maintaining accuracy, and demonstrates its deployment in Meta's recommendation systems.

Contribution

It proposes a joint learning approach for hierarchical indexes using cross-attention and residual quantization, improving retrieval efficiency and inference performance.

Findings

Hierarchical index nodes correspond to high-quality data subsets.

Fine-tuning on these subsets enhances inference performance.

The method is successfully deployed at Meta for billions of users.

Abstract

The increase in data volume, computational resources, and model parameters during training has led to the development of numerous large-scale industrial retrieval models for recommendation tasks. However, effectively and efficiently deploying these large-scale foundational retrieval models remains a critical challenge that has not been fully addressed. Common quick-win solutions for deploying these massive models include relying on offline computations (such as cached user dictionaries) or distilling large models into smaller ones. Yet, both approaches fall short of fully leveraging the representational and inference capabilities of foundational models. In this paper, we explore whether it is possible to learn a hierarchical organization over the memory of foundational retrieval models. Such a hierarchical structure would enable more efficient search by reducing retrieval costs while preserving exactness. To achieve this, we propose jointly learning a hierarchical index using cross-attention and residual quantization for large-scale retrieval models. We also present its real-world deployment at Meta, supporting daily advertisement recommendations for billions of Facebook and Instagram users. Interestingly, we discovered that the intermediate nodes in the learned index correspond to a small set of high-quality data. Fine-tuning the model on this set further improves inference performance, and concretize the concept of "test-time training" within the recommendation system domain. We demonstrate these findings using both internal and public datasets with strong baseline comparisons and hope they contribute to the community's efforts in developing the next generation of foundational retrieval models.