Confidence Ranking for CTR Prediction

TL;DR

This paper introduces Confidence Ranking, a novel loss function for CTR prediction that directly optimizes model logits for metrics like AUC and Accuracy, improving performance in real-world ad systems.

Contribution

The paper proposes a new confidence ranking loss framework that enhances CTR prediction by directly optimizing for specific metrics, with successful deployment in industrial systems.

Findings

Outperforms baseline models on public datasets

Effective in industrial ad systems at JD.com

Improves metrics like AUC and Accuracy

Abstract

Model evolution and constant availability of data are two common phenomena in large-scale real-world machine learning applications, e.g. ads and recommendation systems. To adapt, the real-world system typically retrain with all available data and online learn with recently available data to update the models periodically with the goal of better serving performance. In this paper, we propose a novel framework, named Confidence Ranking, which designs the optimization objective as a ranking function with two different models. Our confidence ranking loss allows direct optimization of the logits output for different convex surrogate functions of metrics, e.g. AUC and Accuracy depending on the target task and dataset. Armed with our proposed methods, our experiments show that the introduction of confidence ranking loss can outperform all baselines on the CTR prediction tasks of public and industrial datasets. This framework has been deployed in the advertisement system of JD.com to serve the main traffic in the fine-rank stage.