Orbit: A Framework for Designing and Evaluating Multi-objective Rankers

TL;DR

Orbit is a framework and interactive system that helps practitioners design and evaluate multi-objective ranking models by focusing on objectives, enabling real-time exploration, and improving decision-making amidst conflicting goals.

Contribution

The paper introduces Orbit, a novel framework and tool for objective-centric design and evaluation of multi-objective rankers, emphasizing stakeholder collaboration and trade-off analysis.

Findings

Supports efficient exploration of design trade-offs

Enhances decision-making with real-time objective analysis

Fosters a broader perspective beyond narrow metrics

Abstract

Machine learning in production needs to balance multiple objectives: This is particularly evident in ranking or recommendation models, where conflicting objectives such as user engagement, satisfaction, diversity, and novelty must be considered at the same time. However, designing multi-objective rankers is inherently a dynamic wicked problem -- there is no single optimal solution, and the needs evolve over time. Effective design requires collaboration between cross-functional teams and careful analysis of a wide range of information. In this work, we introduce Orbit, a conceptual framework for Objective-centric Ranker Building and Iteration. The framework places objectives at the center of the design process, to serve as boundary objects for communication and guide practitioners for design and evaluation. We implement Orbit as an interactive system, which enables stakeholders to interact with objective spaces directly and supports real-time exploration and evaluation of design trade-offs. We evaluate Orbit through a user study involving twelve industry practitioners, showing that it supports efficient design space exploration, leads to more informed decision-making, and enhances awareness of the inherent trade-offs of multiple objectives. Orbit (1) opens up new opportunities of an objective-centric design process for any multi-objective ML models, as well as (2) sheds light on future designs that push practitioners to go beyond a narrow metric-centric or example-centric mindset.