Towards Automated Model Design on Recommender Systems

TL;DR

This paper presents an AutoML approach for designing efficient deep learning-based recommender systems, using a supernet with weight sharing to explore diverse architectures and hardware configurations, leading to state-of-the-art results.

Contribution

It introduces a novel supernet-based AutoML paradigm for joint architecture and hardware co-design in recommender systems, addressing data heterogeneity and complexity.

Findings

Outperforms manually designed and AutoML models on CTR benchmarks.

Achieves 2x FLOPs efficiency and 1.8x energy efficiency improvements.

Delivers 1.5x performance gains in recommender models.

Abstract

The increasing popularity of deep learning models has created new opportunities for developing AI-based recommender systems. Designing recommender systems using deep neural networks requires careful architecture design, and further optimization demands extensive co-design efforts on jointly optimizing model architecture and hardware. Design automation, such as Automated Machine Learning (AutoML), is necessary to fully exploit the potential of recommender model design, including model choices and model-hardware co-design strategies. We introduce a novel paradigm that utilizes weight sharing to explore abundant solution spaces. Our paradigm creates a large supernet to search for optimal architectures and co-design strategies to address the challenges of data multi-modality and heterogeneity in the recommendation domain. From a model perspective, the supernet includes a variety of operators, dense connectivity, and dimension search options. From a co-design perspective, it encompasses versatile Processing-In-Memory (PIM) configurations to produce hardware-efficient models. Our solution space's scale, heterogeneity, and complexity pose several challenges, which we address by proposing various techniques for training and evaluating the supernet. Our crafted models show promising results on three Click-Through Rates (CTR) prediction benchmarks, outperforming both manually designed and AutoML-crafted models with state-of-the-art performance when focusing solely on architecture search. From a co-design perspective, we achieve 2x FLOPs efficiency, 1.8x energy efficiency, and 1.5x performance improvements in recommender models.