Know your population and know your model: Using model-based regression and poststratification to generalize findings beyond the observed sample

TL;DR

This paper extends multilevel regression and poststratification (MRP) to psychology, enabling researchers to generalize findings from non-representative samples by modeling interactions and using population data.

Contribution

It introduces a model-based regression and poststratification approach tailored for psychological research to improve generalizability beyond observed samples.

Findings

Estimated the Big Five Personality Scale distribution using open data.

Demonstrated how MRP can address generalizability in psychology.

Highlighted the potential of combining open data with MRP for replication.

Abstract

Psychology research focuses on interactions, and this has deep implications for inference from non-representative samples. For the goal of estimating average treatment effects, we propose to fit a model allowing treatment to interact with background variables and then average over the distribution of these variables in the population. This can be seen as an extension of multilevel regression and poststratification (MRP), a method used in political science and other areas of survey research, where researchers wish to generalize from a sparse and possibly non-representative sample to the general population. In this paper, we discuss areas where this method can be used in the psychological sciences. We use our method to estimate the norming distribution for the Big Five Personality Scale using open source data. We argue that large open data sources like this and other collaborative data sources can be combined with MRP to help resolve current challenges of generalizability and replication in psychology.