Imitate TheWorld: A Search Engine Simulation Platform

TL;DR

This paper introduces AESim, a simulation platform for search engines that uses adversarial learning and real-world data to better predict online performance and optimize revenue metrics in e-commerce search systems.

Contribution

We develop AESim, a novel simulation platform that integrates real data and adversarial learning to evaluate ranking models more accurately for online performance.

Findings

AESim better predicts online performance than traditional metrics.

AESim can serve as a surrogate for real-world search engine evaluation.

The platform effectively captures user behavior patterns using GAIL.

Abstract

Recent E-commerce applications benefit from the growth of deep learning techniques. However, we notice that many works attempt to maximize business objectives by closely matching offline labels which follow the supervised learning paradigm. This results in models obtain high offline performance in terms of Area Under Curve (AUC) and Normalized Discounted Cumulative Gain (NDCG), but cannot consistently increase the revenue metrics such as purchases amount of users. Towards the issues, we build a simulated search engine AESim that can properly give feedback by a well-trained discriminator for generated pages, as a dynamic dataset. Different from previous simulation platforms which lose connection with the real world, ours depends on the real data in AliExpress Search: we use adversarial learning to generate virtual users and use Generative Adversarial Imitation Learning (GAIL) to capture behavior patterns of users. Our experiments also show AESim can better reflect the online performance of ranking models than classic ranking metrics, implying AESim can play a surrogate of AliExpress Search and evaluate models without going online.