Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation

TL;DR

This paper introduces Hybrid-MST, a novel active sampling strategy combining Bayesian optimization and pairwise preference models to efficiently recover ratings from sparse, noisy data, outperforming existing methods.

Contribution

It proposes a hybrid sampling approach using EIG with GM and MST strategies, enhancing preference aggregation accuracy and efficiency.

Findings

Outperforms state-of-the-art preference aggregation methods.

Effective on both simulated and real-world datasets.

Uses Gaussian-Hermite quadrature for efficient EIG estimation.

Abstract

In this paper we present a hybrid active sampling strategy for pairwise preference aggregation, which aims at recovering the underlying rating of the test candidates from sparse and noisy pairwise labelling. Our method employs Bayesian optimization framework and Bradley-Terry model to construct the utility function, then to obtain the Expected Information Gain (EIG) of each pair. For computational efficiency, Gaussian-Hermite quadrature is used for estimation of EIG. In this work, a hybrid active sampling strategy is proposed, either using Global Maximum (GM) EIG sampling or Minimum Spanning Tree (MST) sampling in each trial, which is determined by the test budget. The proposed method has been validated on both simulated and real-world datasets, where it shows higher preference aggregation ability than the state-of-the-art methods.