Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines

TL;DR

This paper introduces psychometric theory and guidelines for evaluating and developing psychological measurement instruments in behavioral software engineering to improve research quality and understanding of human factors.

Contribution

It provides a comprehensive introduction to psychometric evaluation methods and guidelines tailored for SE researchers, promoting the adoption of psychology-based measurement practices.

Findings

Reviewed psychological literature and standards for testing

Outlined procedures for instrument development and validation

Provided an example of psychometric evaluation in SE

Abstract

A meaningful and deep understanding of the human aspects of software engineering (SE) requires psychological constructs to be considered. Psychology theory can facilitate the systematic and sound development as well as the adoption of instruments (e.g., psychological tests, questionnaires) to assess these constructs. In particular, to ensure high quality, the psychometric properties of instruments need evaluation. In this paper, we provide an introduction to psychometric theory for the evaluation of measurement instruments for SE researchers. We present guidelines that enable using existing instruments and developing new ones adequately. We conducted a comprehensive review of the psychology literature framed by the Standards for Educational and Psychological Testing. We detail activities used when operationalizing new psychological constructs, such as item pooling, item review, pilot testing, item analysis, factor analysis, statistical property of items, reliability, validity, and fairness in testing and test bias. We provide an openly available example of a psychometric evaluation based on our guideline. We hope to encourage a culture change in SE research towards the adoption of established methods from psychology. To improve the quality of behavioral research in SE, studies focusing on introducing, validating, and then using psychometric instruments need to be more common.