PredictionMarketBench: A SWE-bench-Style Framework for Backtesting Trading Agents on Prediction Markets

TL;DR

PredictionMarketBench provides a standardized, realistic benchmarking framework for evaluating trading agents on prediction markets, incorporating historical data replay, fee modeling, and support for both classical and LLM-based strategies.

Contribution

It introduces a comprehensive, reproducible benchmark for prediction market trading agents, integrating data processing, simulation, and agent interface support.

Findings

Naive agents often underperform due to transaction costs.

Fee-aware strategies remain competitive in volatile markets.

Benchmark includes diverse episodes from cryptocurrency, weather, and sports.

Abstract

Prediction markets offer a natural testbed for trading agents: contracts have binary payoffs, prices can be interpreted as probabilities, and realized performance depends critically on market microstructure, fees, and settlement risk. We introduce PredictionMarketBench, a SWE-bench-style benchmark for evaluating algorithmic and LLM-based trading agents on prediction markets via deterministic, event-driven replay of historical limit-order-book and trade data. PredictionMarketBench standardizes (i) episode construction from raw exchange streams (orderbooks, trades, lifecycle, settlement), (ii) an execution-realistic simulator with maker/taker semantics and fee modeling, and (iii) a tool-based agent interface that supports both classical strategies and tool-calling LLM agents with reproducible trajectories. We release four Kalshi-based episodes spanning cryptocurrency, weather, and sports. Baseline results show that naive trading agents can underperform due to transaction costs and settlement losses, while fee-aware algorithmic strategies remain competitive in volatile episodes.