Connecting Humanities and Social Sciences: Applying Language and Speech Technology to Online Panel Surveys

TL;DR

This study investigates the use of speech recognition and transformer-based models to analyze open-ended survey responses, comparing spoken and typed answers to evaluate accuracy and feasibility in social science research.

Contribution

It demonstrates the application of speech technology and transformer models to analyze open-ended survey data, highlighting their potential and limitations.

Findings

ASR errors impact downstream analysis accuracy

Transformer models perform well without target-specific training

Spoken responses can be effectively analyzed with current technology

Abstract

In this paper, we explore the application of language and speech technology to open-ended questions in a Dutch panel survey. In an experimental wave respondents could choose to answer open questions via speech or keyboard. Automatic speech recognition (ASR) was used to process spoken responses. We evaluated answers from these input modalities to investigate differences between spoken and typed answers.We report the errors the ASR system produces and investigate the impact of these errors on downstream analyses. Open-ended questions give more freedom to answer for respondents, but entail a non-trivial amount of work to analyse. We evaluated the feasibility of using transformer-based models (e.g. BERT) to apply sentiment analysis and topic modelling on the answers of open questions. A big advantage of transformer-based models is that they are trained on a large amount of language materials and do not necessarily need training on the target materials. This is especially advantageous for survey data, which does not contain a lot of text materials. We tested the quality of automatic sentiment analysis by comparing automatic labeling with three human raters and tested the robustness of topic modelling by comparing the generated models based on automatic and manually transcribed spoken answers.