# What you see is not what you get: Observed scale score comparisons misestimate true group differences

## TL;DR

This paper shows that using observed scores to compare groups can lead to incorrect estimates of true differences, emphasizing the need for latent variable approaches in social sciences.

## Contribution

The study demonstrates that observed sum scores misestimate true group differences by up to 25%, highlighting the limitations of traditional methods.

## Key findings

- Cohen’s d values based on observed variance misestimate true group differences in 33 out of 70 cases.
- The average misestimation is 25.0% or 0.048 standard deviations.
- Scale reliability (McDonald’s ω) does not predict the effect size discrepancy.

## Abstract

Social sciences of all kinds are interested in latent variables, their measurement, and how they differ between groups. The present study argues the importance of analyzing mean differences between groups using the latent variable approach. Using an open-access repository of widely applied personality questionnaires (N = 999,033), we evaluate the extent to which the commonly used observed sum score is susceptible to measurement error. Our findings show that Cohen’s d values based on the observed variance significantly misestimate the true group difference (based on just the factor score variance) in 33 of the 70 studied cases, and by an average of 25.0% (or 0.048 standard deviations). There was no meaningful relationship between the effect size discrepancy and scale reliability as measured by McDonald’s ω. We discuss the implications of these results and outline concrete steps that applied researchers can take to improve their analyses.

The online version contains supplementary material available at 10.3758/s13428-025-02639-w.