Post-hoc Selection of Pareto-Optimal Solutions in Search and Recommendation

TL;DR

This paper introduces PDU, a novel post-hoc method for selecting the best Pareto-optimal solution in multi-objective IR and RS tasks, effectively considering user preferences and outperforming existing strategies.

Contribution

The paper proposes PDU, a theoretically-justified, post-hoc technique for selecting a single Pareto-optimal solution, incorporating user-specific calibration for better personalization.

Findings

PDU outperforms existing strategies in solution selection.

Calibration enhances the effectiveness of PDU.

The framework is applicable across IR and RS tasks.

Abstract

Information Retrieval (IR) and Recommender Systems (RS) tasks are moving from computing a ranking of final results based on a single metric to multi-objective problems. Solving these problems leads to a set of Pareto-optimal solutions, known as Pareto frontier, in which no objective can be further improved without hurting the others. In principle, all the points on the Pareto frontier are potential candidates to represent the best model selected with respect to the combination of two, or more, metrics. To our knowledge, there are no well-recognized strategies to decide which point should be selected on the frontier. In this paper, we propose a novel, post-hoc, theoretically-justified technique, named "Population Distance from Utopia" (PDU), to identify and select the one-best Pareto-optimal solution from the frontier. In detail, PDU analyzes the distribution of the points by investigating how far each point is from its utopia point (the ideal performance for the objectives). The possibility of considering fine-grained utopia points allows PDU to select solutions tailored to individual user preferences, a novel feature we call "calibration". We compare PDU against existing state-of-the-art strategies through extensive experiments on tasks from both IR and RS. Experimental results show that PDU and combined with calibration notably impact the solution selection. Furthermore, the results show that the proposed framework selects a solution in a principled way, irrespective of its position on the frontier, thus overcoming the limits of other strategies.