The Potential of Using Vision Videos for CrowdRE: Video Comments as a Source of Feedback

TL;DR

This paper investigates the potential of using vision videos on social media as a source of feedback in crowd-based requirements engineering, analyzing comments to assess their relevance and classification performance.

Contribution

It provides an empirical case study demonstrating the high engagement and relevance of video comments for CrowdRE, highlighting the potential of vision videos as feedback sources.

Findings

Over 50% of comments were generated in four days, indicating high engagement.

Comments covered typical feedback topics like feature requests and problem reports.

Machine learning algorithms classified comments effectively, supporting automated analysis.

Abstract

Vision videos are established for soliciting feedback and stimulating discussions in requirements engineering (RE) practices, such as focus groups. Different researchers motivated the transfer of these benefits into crowd-based RE (CrowdRE) by using vision videos on social media platforms. So far, however, little research explored the potential of using vision videos for CrowdRE in detail. In this paper, we analyze and assess this potential, in particular, focusing on video comments as a source of feedback. In a case study, we analyzed 4505 comments on a vision video from YouTube. We found that the video solicited 2770 comments from 2660 viewers in four days. This is more than 50% of all comments the video received in four years. Even though only a certain fraction of these comments are relevant to RE, the relevant comments address typical intentions and topics of user feedback, such as feature request or problem report. Besides the typical user feedback categories, we found more than 300 comments that address the topic safety, which has not appeared in previous analyses of user feedback. In an automated analysis, we compared the performance of three machine learning algorithms on classifying the video comments. Despite certain differences, the algorithms classified the video comments well. Based on these findings, we conclude that the use of vision videos for CrowdRE has a large potential. Despite the preliminary nature of the case study, we are optimistic that vision videos can motivate stakeholders to actively participate in a crowd and solicit numerous of video comments as a valuable source of feedback.