Aggregating Correlated Estimations with (Almost) no Training

TL;DR

This paper introduces new aggregation methods for combining correlated estimation algorithms, demonstrating that maximum likelihood is best with known correlations, while Embedded Voting is preferable with limited data, based on synthetic experiments.

Contribution

It proposes correlation-aware aggregation rules and compares their effectiveness, highlighting the advantages of maximum likelihood and Embedded Voting under different data conditions.

Findings

Maximum likelihood aggregation outperforms naive methods when correlations are known.

Embedded Voting performs well with limited training data.

Synthetic experiments validate the proposed aggregation strategies.

Abstract

Many decision problems cannot be solved exactly and use several estimation algorithms that assign scores to the different available options. The estimation errors can have various correlations, from low (e.g. between two very different approaches) to high (e.g. when using a given algorithm with different hyperparameters). Most aggregation rules would suffer from this diversity of correlations. In this article, we propose different aggregation rules that take correlations into account, and we compare them to naive rules in various experiments based on synthetic data. Our results show that when sufficient information is known about the correlations between errors, a maximum likelihood aggregation should be preferred. Otherwise, typically with limited training data, we recommend a method that we call Embedded Voting (EV).