Using Survey Data to Obtain More Representative Site Samples for Impact Studies

TL;DR

This paper introduces a novel method for selecting more representative impact study sites by leveraging survey data on impact moderators from a representative sample, ensuring balanced site characteristics.

Contribution

It proposes a new approach that uses survey data to set limits on site characteristics, improving sample representativeness when rich impact moderator data are available.

Findings

Method improves sample representativeness in impact evaluations

Enforces limits on site characteristics during selection

Enhances generalizability of impact study results

Abstract

To improve the generalizability of impact evaluations, recent research has examined statistical methods for selecting representative samples of sites. However, these methods rely on having rich data on impact moderators for all sites in the target population. This paper offers a new approach to selecting sites for impact studies when rich data on impact moderators are available, but only from a survey based on a representative sample of the impact study's target population. Survey data are used to (1) estimate the proportion of sites in the population with certain characteristics, and (2) set limits on the number of sites with different characteristics that the sample can include. The Principal Investigator enforces the limits to ensure that certain types of sites are not overrepresented in the final sample. These limits can be layered on top of site selection and recruitment approaches to improve the representativeness of the sample.