TheFragebogen: A Web Browser-based Questionnaire Framework for Scientific Research

TL;DR

TheFragebogen is an open-source, web-based framework designed for conducting QoE research questionnaires, supporting diverse devices, multimedia playback, and behavioral data collection, simplifying experiment replication and deployment.

Contribution

It introduces a versatile, extendable web browser-based questionnaire framework tailored specifically for QoE research, addressing limitations of existing software solutions.

Findings

Supports a broad range of devices via web browsers

Includes graphical scales and free-hand input options

Enables multimedia playback and behavioral data collection

Abstract

Quality of Experience (QoE) typically involves conducting experiments in which stimuli are presented to participants and their judgments as well as behavioral data are collected. Nowadays, many experiments require software for the presentation of stimuli and the data collection from participants. While different software solutions exist, these are not tailored to conduct experiments on QoE. Moreover, replicating experiments or repeating the same experiment in different settings (e. g., laboratory vs. crowdsourcing) can further increase the software complexity. TheFragebogen is an open-source, versatile, extendable software framework for the implementation of questionnaires - especially for research on QoE. Implemented questionnaires can be presented with a state-of-the-art web browser to support a broad range of devices while the use of a web server being optional. Out-of-the-box, TheFragebogen provides graphical exact scales as well as free-hand input, the ability to collect behavioral data, and playback multimedia content.

Tables

Table 1. TABLE I: TheFragebogen technology and links

Software license	MIT License
Programming language	JavaScript (ES5/ES6), CSS3, HTML5
Project web page	http://TheFragebogen.de
Live demos	http://TheFragebogen.de/demo
Software releases	http://TheFragebogen.de/releases
Source code repository	http://TheFragebogen.de/code
Issue tracker	http://TheFragebogen.de/issues

Full text

††publicationid: pubid: XXX-X-XXXX-XXXX-0/19/$31.00 ©2019 IEEE

2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)

I Introduction

In Quality of Experience (QoE) research, questionnaires are a fundamental research instrument to gather data in experiments. In such experiments, participants are presented with a set of stimuli that cover the desired spectrum of degradations, and their perceived quality is captured with questionnaires [1]. In this paper, we present the open-source software framework TheFragebogen that is especially designed (but not limited to) usage in this research domain. TheFragebogen enables implementing questionnaires to be executed by a web browser, which allows to support a broad range of devices (even to run multi-device experiments) and simplifies reproducibility.

Structure. We begin by describing the design of TheFragebogen and its features. To highlight the benefits of TheFragebogen, we present several experiments it was used in. Finally, we present lessons learned and give an outlook on future work.

II TheFragebogen

TheFragebogen has three design principles:

• •

focus on research needs (e. g., scales and multimedia)

• •

extendability & flexibility (e. g., server and serverless use)

• •

robustness & reproducibility (e. g., long-term archival)

In TheFragebogen, questionnaires are implemented as web pages using JavaScript, Cascading Style Sheets (CSS), and Hypertext Markup Language (HTML). For portability, it uses standardized web technologies that are available on a broad range of devices. Further, the text-based nature of these technologies enables using version-control software. While servers can be used in addition, its principle design is serverless. Questionnaires can thus be archived, re-used, or shared, and thus hopefully reduce the effort to reproduce experiments.

There exists already a broad range of (commercial) tools for surveys and questionnaires (e. g., LimeSurvey.org and SurveyJS.io). However, available frameworks commonly focus on commercial needs rather than research needs (e. g., no/limited support for scales and the ability to capture behavioral data) — a gap that we aim to close with TheFragebogen.

II-A Features to simplify conducting experiments

We next describe TheFragebogen’s core features that address the needs of scientific experiments and in combination differentiate it from existing software solutions.

Scales. TheFragebogen provides a set of built-in scales using HTML UI components and also implements several scales that require an exact graphical representation. At the time of this writing, this includes: the NASA task load index (TLX) [2], the visual analogue scale (VAS), and the continuous rating scale for perceived quality [3, p. 19]. This is especially important as replacing graphical scales with non-similar visualizations can have undesired and potentially unknown effects on the questionees’ responses [4]. New graphical scales can be added as scalable vector graphics (SVG).

Free-hand input. TheFragebogen provides support for free-hand input. This enables to capture handwritten responses if a digitizer pen on a tablet computer is used. Using this approach, pen-and-paper questionnaires can be directly recreated without implementing the complete functionality.

Behavioral data. TheFragebogen allows capturing behavioral data (e. g., time to answer a question). This enables, investigating observed anomalies or verify that a participant fulfilled experimental requirements (e. g., in crowdsouring).

Multimedia content. TheFragebogen provides features for presenting multimedia content (i. e., audio and video playback) while capturing stalling and playback statistics.

Remote interaction. For interaction with other technical systems, TheFragebogen provides components to send and receive data via WebSockets as well as Asynchronous JavaScript and XML (AJAX). This enables multi-device experiments (e. g., presenting stimuli and questionnaires on different devices).

Data export. Collected data, incl. questionees’ responses, can be downloaded to the device presenting the questionnaire or uploaded to a web server. As format comma-separated value (CSV) was chosen as it is widely used. Images (e. g., produced by free-hand input) are encoded as Data URIs.

Extensible design. TheFragebogen uses object-oriented programming (OOP) for realizing its components. This simplifies the development and maintenance process since new components can be added based upon existing ones.

Dynamic execution and printability. TheFragebogen enables implementing dynamic questionnaires (i. e., subsequent content is adjusted depending on previous answers). Moreover, questionnaires may be printed to be used as pen-and-paper questionnaires (e. g., as fallback).

Respect for privacy. TheFragebogen enables gathering data while allowing to collect and store the data without sharing it with a third party (e. g., a cloud provider). This is especially important considering that the collected data may be sensitive.

II-B Lifecycle & implementation

A TheFragebogen questionnaire is a single-page application. In this application, the questionnaire is configured using JavaScript (e. g., defining the items and their presentation order). If desired, CSS can be used to adjust the layout and design. On execution, TheFragebogen creates the UI components and manages the state of the questionnaire. Therefore, the use of a web server is optional.

The typical lifecycle of a questionnaire created with TheFragebogen is shown in Figure 1. It consists of the construction phase, the usage phase, and the collection phase. In the construction phase, the questionnaire is designed and prepared for usage. This also includes the implementation, testing, and, if necessary, setup of required infrastructure. Tailoring content to subgroups of participants (e. g., randomization) can be implemented using preprocessor software (e. g., Python or shell scripts). In the usage phase, the questionnaires are distributed to the questionees. In the collection phase, the completed questionnaires are collected and the gathered data is stored for subsequent evaluation. In these two phases, system failure is problematic as this may lead to data loss. This includes implementation errors, compatibility issues with web browser(s), and varying network connectivity.

TheFragebogen follows the paper metaphor, i. e., a paper-based questionnaire is actually a sequence of paper sheets, which each consists of one or more questionnaire items. An item consists, in fact, of a question and a scale. Sheets are represented in TheFragebogen by so-called Screens and items by so-called QuestionnaireItems. A questionnaire consists of one or more Screens while only one is shown at a time. Screens encapsulate specific functionalities, such as presenting items, waiting for some time, or exporting the collected data. A Screen is usually presented in full-screen. The lifecycle of a questionnaire is handled by the ScreenController, which organizes the presentation of all Screens and data consistency.

III Use cases: experiments using TheFragebogen

To highlight TheFragebogen’s suitability and versatility for use in research, we present conducted experiments investigating different aspects of QoE. In these, TheFragebogen was used and they also largely influenced its software design.

III-A Single device experiments

Typical experiments are executed on a single device only (e. g., tablet or computer). Here, stimuli are presented on the same device that captures the participant’s judgments.

Audio stimuli. TheFragebogen was used to assess the perceived quality of preprocessed audio stimuli [5]. Here, a tablet computer and a pair of headphones were used. A tablet computer was chosen as participants could use a digitizer pen to interact with the questionnaire. TheFragebogen was configured to present one stimulus and subsequently present the desired item(s). These included category rating scales and text fields. The later captured a qualitative description of the presented stimuli. Participants answered these in their own hand writing. At the end of the experiment, the collected data was downloaded onto the tablet computer.

Web browsing. Similarly, the perception of the loading delay and load failures of web pages was investigated [6]. For this experiment, TheFragebogen was extended to provide UI components that simulate load delay and load failures of images. This enables to produce these conditions and therefore allows to reproduce the degradations. The stimuli were presented in a randomized order to every participant. Randomization was created in the construction phase using a template questionnaire and a script to generate the randomized questionnaires. By archiving these questionnaires, the experiment for each participant can be reproduced including the presentation order of stimuli.

III-B Experiments using multiple devices

TheFragebogen was also used in experiments requiring a multi-device setup. In these, the questionnaire and stimuli are presented on different devices. This was achieved by using multiple TheFragebogen instances. This approach was used for the assessment of degraded video stimuli on a mobile device while capturing the responses on a tablet computer [7, p. 42ff.]. Participants used the devices alternatingly, as stimuli and items were presented subsequently. In this experiment, both devices were connected to a WiFi hotspot and exchanged the experimental progress via WebSockets. A similar setup was used to investigate the impact of parallel tasks on QoE [8].

TheFragebogen was also used to conduct experiments on conversational speech telephony [7, p. 37ff.]. In these, a pair of participants conducted a series of telephone calls together. For simulation of the telephony connection incl. degradations, the audio processing software TheTelephone111https://github.com/TheTelephone, executed on a separate computer, was used. The participants’ headsets were connected via analogue HiFi cables and microphone amplifiers to this computer. Each participant used a tablet computer running one TheFragebogen instance throughout the experiment. These synchronized their experimental progress incl. presentation of tasks and items as well as informing the audio processing computer about the degradation to be applied.

III-C Remote participation

Field experiments. TheFragebogen was also used in experiments outside of controlled laboratory settings. In a 6 day experiment [7, p. 66], questionnaires were distributed using a web server that could be accessed by the participants with their private computers. On access by a participant, the correct questionnaire was selected depending on his experimental progress. To avoid issues due to varying network conditions all multimedia stimuli were preloaded. After finishing the questionnaire, the collected data was uploaded to the web server. On failure, the data was downloaded to the participant’s computer with the instruction to send it per email to the researchers. For this experiment, participants were required to use Google Chrome limiting the testing effort required.

Crowdsourcing experiments. TheFragebogen was also used in crowdsourcing experiments. For example, in an experiment to assess the influence of network protocols on the perceived web page loading speed [9]. To ensure that every participant rated the same stimuli, the captured load processes were presented as videos. In line with best practices on crowdsourcing [10], filtering mechanisms were realized by capturing behavioral data. For example, to ensure that stimuli were presented completely, focus lost events were captured. Without modification, TheFragebogen’s design enabled to directly repeat the experiment in a laboratory environment to compare the crowdsourcing results with the laboratory results.

III-D Lessons learned

Each of the presented experiments required adding new and refining existing features of TheFragebogen. While using and extending TheFragebogen, we learned to consider practical aspects to avoid failures. First of all, technical complexity should be limited if possible and potential issues while running an experiment should be anticipated (e. g., limited network connectivity, varying web browser). In addition, it is often beneficial to orchestrate experiments using TheFragebogen. This avoids human mistakes, for example, if other systems need to be re-configured within an experiment. With regard to long-term use of implemented questionnaires, it is preferable to use established, wide-spread technologies. Browser-specific functionality or work-in-progess technologies tend to change and then break so far functioning implementations.

IV Conclusion

We presented TheFragebogen, an open-source software framework for creating questionnaires. It is portable, versatile, and extendable as well as out-of-the-box suited for a wide range of use cases in research (e. g., serverless single device questionnaires to crowdsourcing). TheFragebogen evolved by being used in a number of experiments investigating QoE from different aspects. By openly releasing it, we aim to ease future experiments that benefit from the practical experience that went into its software design and its feature set.

We hope TheFragebogen may be one aspect to foster reproducibility of experiments. This could be achieved if researchers openly share their implementations as well as extensions to TheFragebogen. We believe that a shared software solution can be beneficial for the QoE research community, as it promotes exchange of technical expertise and distributes the effort for extending and maintaining the software.

Acknowledgements

Lucas de Araujo, Maxim Spur, and Rodolfo C. Dalapicola contributed to TheFragebogen implementation. Anna Wunderlich, Falk R. Schiffner, Marc Seebode, Marc Halbbrügge, Babak Naderi, and Sebastian Schuck supported TheFragebogen by using and testing it as well as by requesting new features. While working on TheFragebogen Henrique R. Orefice, Lucas de Araujo, and Rodolfo C. Dalapicola were funded by the Ciência sem Fronteiras Alemanha program.