Comparing chatbots to psychometric tests in hiring: reduced social desirability bias, but lower predictive validity

TL;DR

AI chatbots in hiring show less bias but are less accurate than traditional personality tests for predicting job outcomes.

Contribution

Introduces a novel one-question-per-facet approach for evaluating AI chatbots' psychometric properties in hiring.

Findings

Chatbots showed good validity for Extraversion and Conscientiousness but not for Neuroticism, Agreeableness, and Openness.

AI-inferred scores are less affected by social desirability bias compared to traditional tests.

Chatbot scores did not significantly predict real-world hiring outcomes, indicating poor predictive validity.

Abstract

This paper explores the efficacy of AI-driven chatbots in accurately inferring personality traits compared to traditional psychometric tests within a real-world professional hiring context. The study is driven by the increasing integration of AI tools in recruitment processes, which necessitates a deeper understanding of their reliability and validity. Using a quasi-experimental design with propensity score matching, we analysed data from 159 candidates and other professionals from Serbian and Montenegrin regions who completed both traditional psychometric assessments and AI-based personality evaluations based on the Big Five Personality model. A novel one-question-per-facet approach was employed in the chatbot assessments with a goal of enabling more granular analysis of the chatbot’s psychometric properties. The findings indicate that the chatbot demonstrated good structural, substantive, and convergent validity for certain traits, particularly Extraversion and Conscientiousness, but not for Neuroticism, Agreeableness, and Openness. While robust regression confirmed that AI-inferred scores are less susceptible to social desirability bias than traditional tests, they did not significantly predict real-world outcomes, indicating issues with external validity, particularly predictive validity. The results suggest that AI-driven chatbots show promise for identifying certain personality traits and demonstrate resistance to social desirability bias. This paper contributes to the emerging field of AI and psychometrics by offering insights into the potential and limitations of AI tools in professional selection, while developing an approach for refining psychometric properties of AI-driven assessments.

Figures