A prototype hybrid prediction market for estimating replicability of published work

TL;DR

This paper introduces a prototype hybrid prediction market combining human and AI inputs to estimate the replicability of published research, demonstrating potential for improved scientific validation.

Contribution

It presents a novel hybrid prediction market framework integrating human and AI agents for assessing research replicability, expanding on artificial prediction market algorithms.

Findings

Successful implementation of a prototype hybrid market

Insights into human-AI collaboration dynamics

Identification of challenges and opportunities in prediction markets

Abstract

We present a prototype hybrid prediction market and demonstrate the avenue it represents for meaningful human-AI collaboration. We build on prior work proposing artificial prediction markets as a novel machine-learning algorithm. In an artificial prediction market, trained AI agents buy and sell outcomes of future events. Classification decisions can be framed as outcomes of future events, and accordingly, the price of an asset corresponding to a given classification outcome can be taken as a proxy for the confidence of the system in that decision. By embedding human participants in these markets alongside bot traders, we can bring together insights from both. In this paper, we detail pilot studies with prototype hybrid markets for the prediction of replication study outcomes. We highlight challenges and opportunities, share insights from semi-structured interviews with hybrid market participants, and outline a vision for ongoing and future work.