Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction

TL;DR

This paper introduces AGE, an adversarial gradient-based exploration method for deep CTR prediction that considers exploration's impact on model training, leading to improved online recommendation performance.

Contribution

The paper proposes a novel exploration strategy that simulates model updates via adversarial perturbations, integrating a dynamic gating mechanism for resource-efficient exploration in large-scale recommender systems.

Findings

Significant improvements in top-line metrics on a display advertising platform.

Effective modeling of exploration's influence on training process.

Validated through extensive ablation studies on academic datasets.

Abstract

Exploration-Exploitation (E{\&}E) algorithms are commonly adopted to deal with the feedback-loop issue in large-scale online recommender systems. Most of existing studies believe that high uncertainty can be a good indicator of potential reward, and thus primarily focus on the estimation of model uncertainty. We argue that such an approach overlooks the subsequent effect of exploration on model training. From the perspective of online learning, the adoption of an exploration strategy would also affect the collecting of training data, which further influences model learning. To understand the interaction between exploration and training, we design a Pseudo-Exploration module that simulates the model updating process after a certain item is explored and the corresponding feedback is received. We further show that such a process is equivalent to adding an adversarial perturbation to the model input, and thereby name our proposed approach as an the Adversarial Gradient Driven Exploration (AGE). For production deployment, we propose a dynamic gating unit to pre-determine the utility of an exploration. This enables us to utilize the limited amount of resources for exploration, and avoid wasting pageview resources on ineffective exploration. The effectiveness of AGE was firstly examined through an extensive number of ablation studies on an academic dataset. Meanwhile, AGE has also been deployed to one of the world-leading display advertising platforms, and we observe significant improvements on various top-line evaluation metrics.