Test-time Collective Prediction

TL;DR

This paper introduces a decentralized test-time prediction method for multiple agents with private data, inspired by social consensus, which improves accuracy over traditional averaging techniques without data sharing.

Contribution

It proposes a novel decentralized mechanism for collective predictions that converges to inverse MSE weighting and includes a Jackknife-based error estimation method.

Findings

Outperforms classical model averaging in accuracy.

Effectively combines models with varying quality.

Achieves better results than weighted averaging with validation data.

Abstract

An increasingly common setting in machine learning involves multiple parties, each with their own data, who want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents to make better predictions than they would individually, but may not be willing to release their data or model parameters. In this work, we explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model without relying on external validation, model retraining, or data pooling. Our approach takes inspiration from the literature in social science on human consensus-making. We analyze our mechanism theoretically, showing that it converges to inverse meansquared-error (MSE) weighting in the large-sample limit. To compute error bars on the collective predictions we propose a decentralized Jackknife procedure that evaluates the sensitivity of our mechanism to a single agent's prediction. Empirically, we demonstrate that our scheme effectively combines models with differing quality across the input space. The proposed consensus prediction achieves significant gains over classical model averaging, and even outperforms weighted averaging schemes that have access to additional validation data.