Multi-objective Learning to Rank by Model Distillation

TL;DR

This paper introduces a distillation-based multi-objective ranking method for online marketplaces that improves primary and secondary objectives, enhances stability, and efficiently incorporates ad-hoc goals.

Contribution

The paper proposes a novel distillation approach for multi-objective learning to rank, addressing industry challenges like parameter tuning, data sparsity, and ad-hoc objective integration.

Findings

Significantly improves primary ranking objectives.

Effectively meets secondary objectives constraints.

Enables efficient integration of ad-hoc non-differentiable objectives.

Abstract

In online marketplaces, search ranking's objective is not only to purchase or conversion (primary objective), but to also the purchase outcomes(secondary objectives), e.g. order cancellation(or return), review rating, customer service inquiries, platform long term growth. Multi-objective learning to rank has been widely studied to balance primary and secondary objectives. But traditional approaches in industry face some challenges including expensive parameter tuning leads to sub-optimal solution, suffering from imbalanced data sparsity issue, and being not compatible with ad-hoc objective. In this paper, we propose a distillation-based ranking solution for multi-objective ranking, which optimizes the end-to-end ranking system at Airbnb across multiple ranking models on different objectives along with various considerations to optimize training and serving efficiency to meet industry standards. We found it performs much better than traditional approaches, it doesn't only significantly increases primary objective by a large margin but also meet secondary objectives constraints and improve model stability. We also demonstrated the proposed system could be further simplified by model self-distillation. Besides this, we did additional simulations to show that this approach could also help us efficiently inject ad-hoc non-differentiable business objective into the ranking system while enabling us to balance our optimization objectives.