A nonmanipulable test

TL;DR

This paper introduces a new statistical test that controls for type I error and is resistant to manipulation by strategic experts, addressing a key limitation of existing tests.

Contribution

The paper presents a novel test that simultaneously controls for type I error and is nonmanipulable, unlike many existing calibration tests.

Findings

The proposed test is nonmanipulable.

It effectively controls for type I error.

It outperforms existing tests in strategic settings.

Abstract

A test is said to control for type I error if it is unlikely to reject the data-generating process. However, if it is possible to produce stochastic processes at random such that, for all possible future realizations of the data, the selected process is unlikely to be rejected, then the test is said to be manipulable. So, a manipulable test has essentially no capacity to reject a strategic expert. Many tests proposed in the existing literature, including calibration tests, control for type I error but are manipulable. We construct a test that controls for type I error and is nonmanipulable.