Can LLMs Help Decentralized Dispute Arbitration? A Case Study of UMA-Resolved Markets on Polymarket

TL;DR

This paper investigates the potential of large language models (LLMs) to assist in dispute resolution and prediction in Web3 prediction markets, specifically Polymarket, finding they match UMA's resolutions well but cannot predict disputes beforehand.

Contribution

It introduces the application of LLMs to evaluate dispute outcomes and predict future disputes in decentralized markets, a novel approach in this domain.

Findings

LLMs achieve 89.58% agreement with UMA's dispute resolutions.

LLMs cannot reliably predict which events will face disputes in advance.

Web-enabled LLMs show strong stability once a dispute is initiated.

Abstract

Web3 prediction markets, exemplified by Polymarket, have gained prominence for leveraging collective intelligence to forecast a wide range of social, political, and sports events. However, among the thousands of prediction market events, consensus disputes still arise due to imperfections in market mechanisms. On Polymarket alone, the trading volume involving disputed events has reached $972,370,804.71, underscoring the critical need for objective and efficient dispute resolution. In this study, we introduce large language models (LLMs) to: (1) evaluate whether web-enabled LLMs can reproduce the decision quality of UMA's on-chain voting process once a dispute has been raised, and (2) predict, based on event rules, which market events are likely to face future disputes before they occur. Our findings show that LLMs are unable to reliably predict which events will become disputed in advance; however, once a dispute is initiated, web-enabled LLMs achieve 89.58% agreement with UMA's final resolutions and demonstrate strong stability.