Crowdsourcing: A Framework for Usability Evaluation

TL;DR

This paper presents a framework for using crowdsourcing as a cost-effective and efficient alternative to expert heuristic evaluation in software usability testing, validated through experiments and a case study.

Contribution

It introduces a novel framework for crowd usability inspection that matches expert evaluations and demonstrates its effectiveness through empirical experiments and a case study.

Findings

Crowdsourced usability inspections can match experts in identifying issues.

The framework enables effective usability evaluations with 3-5 novice inspectors.

Crowdsourcing reduces cost and time compared to traditional expert evaluations.

Abstract

Objective: This research explores using crowdsourcing for software usability evaluation. Background: Usability studies are essential for designing user-friendly software, but traditional methods are often costly and time-consuming. Crowdsourcing offers a quicker, cost-effective alternative for remote usability evaluation, though ensuring quality feedback remains a challenge. Method: A systematic mapping study was conducted to review current usability evaluation research. Subsequently, multi-experiments were performed, comparing novice crowd usability inspectors to experts using expert heuristic evaluation as a benchmark. These results were used to create and validate a framework for crowd usability inspection through a case study. Results: The mapping study identified expert heuristic evaluation as a prevalent method, especially for websites. Experimental findings showed that novice crowd usability inspections, guided by expert heuristics, can match experts in identifying usability issues in content, quality, severity, and time efficiency. The case study demonstrated that the framework allows effective usability inspections, leading to successful software redesigns. Iterations of 3-5 novice inspections effectively resolved key usability issues within three cycles. Conclusion: Crowdsourcing is an effective alternative to expert heuristic evaluation for usability assessment. The proposed framework for crowd usability inspection is a viable solution for budget-constrained software companies. Keywords: crowdsourcing, crowd usability evaluation, expert heuristic evaluation, framework.