How many federal employees are not satisfied? Using response times to estimate population proportions under the survey variable cause model

TL;DR

This paper introduces a novel statistical model using response times to better estimate population proportions in surveys affected by causal participation biases, specifically applied to federal employee satisfaction data.

Contribution

It develops a new approach based on survival analysis techniques under the survey variable cause model, addressing nonresponse bias beyond traditional methods.

Findings

Sample overestimates dissatisfaction among federal employees.

Non-satisfied employees are more likely to participate, biasing results.

Traditional adjustments like poststratification are insufficient to correct this bias.

Abstract

We propose a statistical model to estimate population proportions under the survey variable cause model (Groves 2006), the setting in which the characteristic measured by the survey has a direct causal effect on survey participation. For example, we estimate employee satisfaction from a survey in which the decision of an employee to participate depends on their satisfaction. We model the time at which a respondent 'arrives' to take the survey, leveraging results from the counting processes literature that has been developed to analyze similar problems with survival data. Our approach is particularly useful for nonresponse bias analysis because it relies on different assumptions than traditional adjustments such as poststratification, which assumes the common cause model, the setting in which external factors explain the characteristic measured by the survey and participation. Our motivation is the Federal Employee Viewpoint Survey, which asks federal employees whether they are satisfied with their work organization. Our model suggests that the sample proportion overestimates the proportion of federal employees that are not satisfied with their work organization even after adjustment by poststratification. Employees that are not satisfied likely select into the survey, and this selection cannot be explained by personal characteristics like race, gender, and occupation or work-place characteristics like agency, unit, and location.