IMO$^3$: Interactive Multi-Objective Off-Policy Optimization

TL;DR

IMO$^3$ is a novel interactive off-policy optimization method that efficiently finds near-optimal policies for unknown multi-objective functions by engaging with system designers, reducing reliance on costly online testing.

Contribution

The paper introduces IMO$^3$, a new interactive off-policy approach for multi-objective optimization with unknown objectives, providing theoretical guarantees and empirical validation.

Findings

IMO$^3$ effectively identifies near-optimal policies with high probability.

The method reduces the need for costly online A/B testing in industry.

Empirical results show strong performance across multiple optimization problems.

Abstract

Most real-world optimization problems have multiple objectives. A system designer needs to find a policy that trades off these objectives to reach a desired operating point. This problem has been studied extensively in the setting of known objective functions. We consider a more practical but challenging setting of unknown objective functions. In industry, this problem is mostly approached with online A/B testing, which is often costly and inefficient. As an alternative, we propose interactive multi-objective off-policy optimization (IMO$^3$). The key idea in our approach is to interact with a system designer using policies evaluated in an off-policy fashion to uncover which policy maximizes her unknown utility function. We theoretically show that IMO$^3$ identifies a near-optimal policy with high probability, depending on the amount of feedback from the designer and training data for off-policy estimation. We demonstrate its effectiveness empirically on multiple multi-objective optimization problems.