Identifying good forecasters via adaptive cognitive tests

TL;DR

This paper develops adaptive cognitive tests based on item response models to efficiently assess forecasting skill levels, enabling quick, real-time evaluation of forecasters' performance potential.

Contribution

It introduces a novel adaptive testing approach that optimizes cognitive test selection for forecasting ability assessment, reducing testing time and improving predictive accuracy.

Findings

Selected tests are highly predictive of forecasting performance.

Adaptive tests significantly reduce assessment time.

Scores correlate strongly with out-of-sample forecasting accuracy.

Abstract

Assessing forecasting performance is a time intensive activity, often requiring months or years before we know whether or not the reported forecasts were accurate. Cognitive tests can be quickly administered and are predictive of forecasting performance, but it is unclear which and how many tests are optimal. In this study, we develop adaptive cognitive tests that optimize the selection and efficiency of cognitive tests to assess forecasters of different skill levels. The tests are based on item response models and the adaptive testing procedures commonly used in educational testing. We show how the procedures can select highly informative cognitive tests from a larger battery of tests, thereby reducing the time taken to administer the tests. We use a second, independent dataset to show that the selected tests yield scores that are highly related to out-of-sample forecasting performance. The approach enables real-time, adaptive testing, providing immediate insights into forecasting talent in practical contexts.