NASRec: Weight Sharing Neural Architecture Search for Recommender Systems

TL;DR

NASRec introduces a weight sharing neural architecture search method tailored for recommender systems, efficiently exploring diverse architectures within a large supernet to improve CTR prediction performance.

Contribution

It presents a novel NAS framework for recommendation systems that handles heterogeneity and multi-modality, with techniques to improve training efficiency and model quality.

Findings

NASRecNet outperforms manually designed models on CTR benchmarks.

The proposed methods improve training efficiency and model stability.

State-of-the-art results achieved on multiple datasets.

Abstract

The rise of deep neural networks offers new opportunities in optimizing recommender systems. However, optimizing recommender systems using deep neural networks requires delicate architecture fabrication. We propose NASRec, a paradigm that trains a single supernet and efficiently produces abundant models/sub-architectures by weight sharing. To overcome the data multi-modality and architecture heterogeneity challenges in the recommendation domain, NASRec establishes a large supernet (i.e., search space) to search the full architectures. The supernet incorporates versatile choice of operators and dense connectivity to minimize human efforts for finding priors. The scale and heterogeneity in NASRec impose several challenges, such as training inefficiency, operator-imbalance, and degraded rank correlation. We tackle these challenges by proposing single-operator any-connection sampling, operator-balancing interaction modules, and post-training fine-tuning. Our crafted models, NASRecNet, show promising results on three Click-Through Rates (CTR) prediction benchmarks, indicating that NASRec outperforms both manually designed models and existing NAS methods with state-of-the-art performance. Our work is publicly available at https://github.com/facebookresearch/NasRec.