Can Transformer Language Models Predict Psychometric Properties?

TL;DR

This study investigates whether transformer language models can predict the psychometric properties of linguistic test items by comparing their responses to human responses, revealing similarities and differences in reasoning.

Contribution

It introduces a method to evaluate LMs' ability to predict psychometric properties of test items, bridging NLP and psychometrics in a novel way.

Findings

Transformers predict psychometric properties well in some categories.

Transformers perform poorly in predicting properties in other categories.

Insights into similarities and differences between human and LM reasoning.

Abstract

Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired "commonsense" competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformer and non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformer-based LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning.