TL;DR
This paper introduces a framework to evaluate how different collaboration modes affect the outcomes of NLP-assisted qualitative research tools, analyzing consistency, cohesiveness, and correctness.
Contribution
It presents a novel evaluation framework for assessing the impact of collaboration settings on qualitative data analysis tools.
Findings
Synchronous and asynchronous collaboration produce different research outcomes.
The framework reveals variations in consistency, cohesiveness, and correctness based on collaboration mode.
Empirical analysis across three NLP tools demonstrates the framework's effectiveness.
Abstract
NLP-assisted solutions to support qualitative data analysis have gained considerable traction. However, no unified evaluation framework exists which can account for the many different settings in which qualitative researchers may employ them. In this paper, we propose a framework to evaluate the way collaboration settings may produce different research outcomes across a variety of interactive systems. Specifically, we study the impact of synchronous vs. asynchronous collaboration using three different NLP-assisted qualitative research tools and present a comprehensive analysis of the differences in the consistency, cohesiveness, and correctness of their outcomes.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
