Investigating Mode Effects in Interviewer Variances Using Two Representative Multi-mode Surveys

TL;DR

This study investigates whether interviewer variances differ across modes in mixed-mode surveys, revealing mode-related differences in variances for sensitive questions and highlighting the importance of considering mode effects in survey design.

Contribution

It provides empirical evidence on mode effects in interviewer variances across two large-scale surveys, using respondent covariates to address design limitations.

Findings

Significant interviewer variance differences in Arab Barometer for some items.

Larger variances in face-to-face mode for sensitive questions.

Simulation results on power to detect mode effects.

Abstract

This study examines whether interviewer variances remain consistent across different modes in mixed-mode studies, using data from two distinct designs. In the first design, when interviewers are responsible for either face-to-face or telephone mode, we examine whether there are mode differences in interviewer variances for 1) sensitive political questions, 2) international items, 3) and item missing indicators on international items, using the Arab Barometer wave 6 Jordan data. In the second design, we draw on Health and Retirement Study (HRS) 2016 core survey data to examine the question on three topics when interviewers are responsible for both modes. The topics cover 1) the CESD depression scale, 2) interviewer observations, and 3) the physical activity scale. To account for the lack of interpenetrated designs in both data sources, we include respondent-level covariates in our models. We find significant differences in interviewer variances on one item (twelve items in total) in the Arab Barometer study; whereas for HRS, the results are three out of eighteen. Overall, we find the magnitude of the interviewer variances larger in FTF than TEL on sensitive items. We conduct simulations to understand the power to detect mode effects in the typically modest interviewer sample sizes.