Promises and Perils of Inferring Personality on GitHub

TL;DR

This study evaluates the accuracy of text-based personality inference methods on GitHub data, revealing significant errors and proposing improvements to enhance their reliability in software engineering contexts.

Contribution

It compares three popular personality inference tests against ground truth on GitHub, highlighting their limitations and suggesting ways to improve their performance.

Findings

Average error rate of 41% in personality inference from GitHub data

Potential to reduce error rate by up to 36% with recommended methods

Current solutions are far from perfect in inferring developer personalities

Abstract

Personality plays a pivotal role in our understanding of human actions and behavior. Today, the applications of personality are widespread, built on the solutions from psychology to infer personality. In software engineering, for instance, one widely used solution to infer personality uses textual communication data. As studies on personality in software engineering continue to grow, it is imperative to understand the performance of these solutions. This paper compares the inferential ability of three widely studied text-based personality tests against each other and the ground truth on GitHub. We explore the challenges and potential solutions to improve the inferential ability of personality tests. Our study shows that solutions for inferring personality are far from being perfect. Software engineering communications data can infer individual developer personality with an average error rate of 41%. In the best case, the error rate can be reduced up to 36% by following our recommendations.