Open vs Closed-ended questions in attitudinal surveys -- comparing, combining, and interpreting using natural language processing

TL;DR

This paper compares open- and closed-ended survey questions using Topic Modeling to analyze attitudes toward autonomous vehicles, proposing a flexible framework allowing respondents to choose their preferred question type, with findings favoring closed-ended responses for modeling accuracy.

Contribution

It introduces a novel framework enabling respondents to select question types and demonstrates the effectiveness of Topic Modeling for open-ended responses in travel attitude surveys.

Findings

Topic Modeling effectively extracts information from open-ended responses.

Models based on closed-ended responses outperform those from open-ended responses.

The proposed flexible framework improves survey adaptability and modeling performance.

Abstract

To improve the traveling experience, researchers have been analyzing the role of attitudes in travel behavior modeling. Although most researchers use closed-ended surveys, the appropriate method to measure attitudes is debatable. Topic Modeling could significantly reduce the time to extract information from open-ended responses and eliminate subjective bias, thereby alleviating analyst concerns. Our research uses Topic Modeling to extract information from open-ended questions and compare its performance with closed-ended responses. Furthermore, some respondents might prefer answering questions using their preferred questionnaire type. So, we propose a modeling framework that allows respondents to use their preferred questionnaire type to answer the survey and enable analysts to use the modeling frameworks of their choice to predict behavior. We demonstrate this using a dataset collected from the USA that measures the intention to use Autonomous Vehicles for commute trips. Respondents were presented with alternative questionnaire versions (open- and closed- ended). Since our objective was also to compare the performance of alternative questionnaire versions, the survey was designed to eliminate influences resulting from statements, behavioral framework, and the choice experiment. Results indicate the suitability of using Topic Modeling to extract information from open-ended responses; however, the models estimated using the closed-ended questions perform better compared to them. Besides, the proposed model performs better compared to the models used currently. Furthermore, our proposed framework will allow respondents to choose the questionnaire type to answer, which could be particularly beneficial to them when using voice-based surveys.