Modeling dependent group judgments: A computational model of sequential collaboration

TL;DR

This paper introduces a computational model to explain how people collaboratively improve judgments in sequential online collaborations, showing they can be as accurate as group averages.

Contribution

A novel computational model formalizing the cognitive processes of sequential collaboration and dependent group judgments.

Findings

Sequential collaboration leads to less frequent and smaller changes over time.

Judgments at the end of a chain are as accurate as aggregated independent judgments.

Expertise drives changes in judgment accuracy and magnitude in sequential chains.

Abstract

Sequential collaboration describes the incremental process of contributing to online collaborative projects such as Wikipedia and OpenStreetMap. After a first contributor creates an initial entry, subsequent contributors create a sequential chain by deciding whether to adjust or maintain the latest entry which is updated if they decide to make changes. Sequential collaboration has recently been examined as a method for eliciting numerical group judgments. It was shown that in a sequential chain, changes become less frequent and smaller, while judgments become more accurate. Judgments at the end of a sequential chain are similarly accurate and in some cases even more accurate than aggregated independent judgments (wisdom of crowds). This is at least partly due to sequential collaboration allowing contributors to contribute according to their expertise by selectively adjusting judgments. However, there is no formal theory of sequential collaboration. We developed a computational model that formalizes the cognitive processes underlying sequential collaboration. It allows modeling both sequential collaboration and independent judgments, which are used as a benchmark for the performance of sequential collaboration. The model is based on internal distributions of plausible judgments that contributors use to evaluate the plausibility of presented judgments and to provide new judgments. It incorporates individuals’ expertise and tendency to adjust presented judgments as well as item difficulty and the effects of the presented judgment on subsequent judgment formation. The model is consistent with previous empirical findings on change probability, change magnitude, and judgment accuracy incorporating expertise as a driving factor of these effects. Moreover, new predictions for long sequential chains were confirmed by an empirical study. Above and beyond sequential collaboration the model establishes an initial theoretical framework for further research on dependent judgments.

Figures