Uncovering the Data-Related Limits of Human Reasoning Research: An Analysis based on Recommender Systems

TL;DR

This paper explores how data quality issues, such as noise and inconsistencies, limit progress in modeling human reasoning, and proposes data-driven methods like recommender systems to improve understanding and prediction of human responses.

Contribution

It introduces a novel data-centric approach using collaborative filtering to analyze reasoning data, revealing noise levels, performance bounds, and a new paradigm for modeling individual human reasoning.

Findings

Identified the noise levels in human reasoning data.

Found an upper-bound in model performance indicating a need for new modeling approaches.

Demonstrated the potential of recommender systems for personalized reasoning prediction.

Abstract

Understanding the fundamentals of human reasoning is central to the development of any system built to closely interact with humans. Cognitive science pursues the goal of modeling human-like intelligence from a theory-driven perspective with a strong focus on explainability. Syllogistic reasoning as one of the core domains of human reasoning research has seen a surge of computational models being developed over the last years. However, recent analyses of models' predictive performances revealed a stagnation in improvement. We believe that most of the problems encountered in cognitive science are not due to the specific models that have been developed but can be traced back to the peculiarities of behavioral data instead. Therefore, we investigate potential data-related reasons for the problems in human reasoning research by comparing model performances on human and artificially generated datasets. In particular, we apply collaborative filtering recommenders to investigate the adversarial effects of inconsistencies and noise in data and illustrate the potential for data-driven methods in a field of research predominantly concerned with gaining high-level theoretical insight into a domain. Our work (i) provides insight into the levels of noise to be expected from human responses in reasoning data, (ii) uncovers evidence for an upper-bound of performance that is close to being reached urging for an extension of the modeling task, and (iii) introduces the tools and presents initial results to pioneer a new paradigm for investigating and modeling reasoning focusing on predicting responses for individual human reasoners.