AIF: Asynchronous Inference Framework for Cost-Effective Pre-Ranking

TL;DR

AIF is an asynchronous inference framework designed to improve the efficiency and reduce latency in pre-ranking models for recommendation systems by decoupling and parallelizing computations.

Contribution

The paper introduces AIF, a novel asynchronous architecture that reorganizes inference to decouple interaction-independent components, enhancing efficiency and enabling better model design.

Findings

Significant latency reduction in pre-ranking tasks.

Improved computational efficiency allowing richer feature sets.

Successful deployment in Taobao's advertising system.

Abstract

In industrial recommendation systems, pre-ranking models based on deep neural networks (DNNs) commonly adopt a sequential execution framework: feature fetching and model forward computation are triggered only after receiving candidates from the upstream retrieval stage. This design introduces inherent bottlenecks, including redundant computations of identical users/items and increased latency due to strictly sequential operations, which jointly constrain the model's capacity and system efficiency. To address these limitations, we propose the Asynchronous Inference Framework (AIF), a cost-effective computational architecture that decouples interaction-independent components, those operating within a single user or item, from real-time prediction. AIF reorganizes the model inference process by performing user-side computations in parallel with the retrieval stage and conducting item-side computations in a nearline manner. This means that interaction-independent components are calculated just once and completed before the real-time prediction phase of the pre-ranking stage. As a result, AIF enhances computational efficiency and reduces latency, freeing up resources to significantly improve the feature set and model architecture of interaction-independent components. Moreover, we delve into model design within the AIF framework, employing approximated methods for interaction-dependent components in online real-time predictions. By co-designing both the framework and the model, our solution achieves notable performance gains without significantly increasing computational and latency costs. This has enabled the successful deployment of AIF in the Taobao display advertising system.